[Swift-commit] cog r3831
swift at ci.uchicago.edu
swift at ci.uchicago.edu
Thu Nov 14 13:30:24 CST 2013
------------------------------------------------------------------------
r3831 | yadunand | 2013-11-14 13:29:00 -0600 (Thu, 14 Nov 2013) | 3 lines
Minor changes
------------------------------------------------------------------------
Index: modules/provider-coaster-c-client/tests/coasters/coaster-service.conf
===================================================================
--- modules/provider-coaster-c-client/tests/coasters/coaster-service.conf (revision 3830)
+++ modules/provider-coaster-c-client/tests/coasters/coaster-service.conf (working copy)
@@ -12,7 +12,7 @@
export WORKER_USERNAME=$USER
# Worker host names for ssh
-export WORKER_HOSTS="crush.mcs.anl.gov grind.mcs.anl.gov"
+export WORKER_HOSTS="crush.mcs.anl.gov grind.mcs.anl.gov steamroller.mcs.anl.gov crank.mcs.anl.gov thwomp.mcs.anl.gov"
# Directory to keep log files, relative to working directory when launching start-coaster-service
export LOG_DIR=logs
Index: modules/provider-coaster-c-client/tests/coasters/start-coaster-service.log
===================================================================
--- modules/provider-coaster-c-client/tests/coasters/start-coaster-service.log (revision 3830)
+++ modules/provider-coaster-c-client/tests/coasters/start-coaster-service.log (working copy)
@@ -1,129118 +0,0 @@
-Running /homes/yadunand/swift-trunk/cog/modules/swift/dist/swift-svn/bin/coaster-service -nosec -portfile /tmp/tmp.bWAGiTa8uX -localportfile /tmp/tmp.IumeBPaF2V -passive
-Switching log to: cps-2013-11-05_20-20-02.log
-2013-11-05 20:20:02,502-0600 WARN CoasterPersistentService Switching log to: cps-2013-11-05_20-20-02.log
-Local contacts: [http://140.221.8.81:45138]
-2013-11-05 20:20:02,528-0600 INFO Settings Local contacts: [http://140.221.8.81:45138]
-Starting... id=1105-2008020
-2013-11-05 20:20:02,541-0600 INFO BlockQueueProcessor Starting... id=1105-2008020
-Started local service: http://140.221.8.81:45138
-2013-11-05 20:20:02,542-0600 INFO CoasterService Started local service: http://140.221.8.81:45138
-Started coaster service: http://140.221.8.81:45699
-2013-11-05 20:20:02,542-0600 INFO CoasterService Started coaster service: http://140.221.8.81:45699
-Started coaster service: http://140.221.8.81:45699
-Worker connection URL: http://140.221.8.81:45138
-Running ssh yadunand at crush.mcs.anl.gov mkdir -p /homes/yadunand
-Running scp /homes/yadunand/swift-trunk/cog/modules/swift/dist/swift-svn/bin/worker.pl yadunand at crush.mcs.anl.gov:/homes/yadunand
-Running ssh yadunand at crush.mcs.anl.gov WORKER_LOGGING_LEVEL=DEBUG /homes/yadunand/worker.pl http://140.221.8.81:45138 crush.mcs.anl.gov /homes/yadunand/logs
-Running ssh yadunand at grind.mcs.anl.gov mkdir -p /homes/yadunand
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-05 20:20:05,243-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-Channel started: TCPChannel [type: server, contact: 140.221.8.62:51960]
-2013-11-05 20:20:05,248-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 140.221.8.62:51960]
-Received registration: blockid = crush.mcs.anl.gov, url = crush
-2013-11-05 20:20:05,253-0600 INFO LocalTCPService Received registration: blockid = crush.mcs.anl.gov, url = crush
-MetaChannel [context: worker-1, boundTo: null] binding to TCPChannel [type: server, contact: 140.221.8.62:51960]
-2013-11-05 20:20:05,264-0600 INFO MetaChannel MetaChannel [context: worker-1, boundTo: null] binding to TCPChannel [type: server, contact: 140.221.8.62:51960]
-2013-11-05 20:20:05,267-0600 DEBUG Cpu worker started: block=crush.mcs.anl.gov host=crush id=0
-2013-11-05 20:20:05,268-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-Started CPU 0:1383704405s
-2013-11-05 20:20:05,269-0600 INFO Block Started CPU 0:1383704405s
-Started worker crush.mcs.anl.gov:000000
-2013-11-05 20:20:05,269-0600 INFO Block Started worker crush.mcs.anl.gov:000000
-2013-11-05 20:20:05,269-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-Using NIO sender for TCPChannel [type: server, contact: crush.mcs.anl.gov-000000]
-2013-11-05 20:20:05,269-0600 INFO AbstractStreamCoasterChannel Using NIO sender for TCPChannel [type: server, contact: crush.mcs.anl.gov-000000]
-2013-11-05 20:20:05,270-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-Handler(tag: 7420, REGISTER) unregistering (send)
-2013-11-05 20:20:05,273-0600 INFO RequestHandler Handler(tag: 7420, REGISTER) unregistering (send)
-Running scp /homes/yadunand/swift-trunk/cog/modules/swift/dist/swift-svn/bin/worker.pl yadunand at grind.mcs.anl.gov:/homes/yadunand
-Running ssh yadunand at grind.mcs.anl.gov WORKER_LOGGING_LEVEL=DEBUG /homes/yadunand/worker.pl http://140.221.8.81:45138 grind.mcs.anl.gov /homes/yadunand/logs
-Adjusting buffer size to 8650752 for 2 sockets
-2013-11-05 20:20:06,395-0600 INFO TCPBufferManager Adjusting buffer size to 8650752 for 2 sockets
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-05 20:20:06,395-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-Channel started: TCPChannel [type: server, contact: 140.221.8.74:48379]
-2013-11-05 20:20:06,395-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 140.221.8.74:48379]
-Received registration: blockid = grind.mcs.anl.gov, url = grind
-2013-11-05 20:20:06,396-0600 INFO LocalTCPService Received registration: blockid = grind.mcs.anl.gov, url = grind
-MetaChannel [context: worker-2, boundTo: null] binding to TCPChannel [type: server, contact: 140.221.8.74:48379]
-2013-11-05 20:20:06,396-0600 INFO MetaChannel MetaChannel [context: worker-2, boundTo: null] binding to TCPChannel [type: server, contact: 140.221.8.74:48379]
-2013-11-05 20:20:06,396-0600 DEBUG Cpu worker started: block=grind.mcs.anl.gov host=grind id=0
-2013-11-05 20:20:06,396-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Started CPU 0:1383704406s
-2013-11-05 20:20:06,396-0600 INFO Block Started CPU 0:1383704406s
-2013-11-05 20:20:06,396-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Started worker grind.mcs.anl.gov:000000
-2013-11-05 20:20:06,396-0600 INFO Block Started worker grind.mcs.anl.gov:000000
-2013-11-05 20:20:06,397-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-Handler(tag: 7297, REGISTER) unregistering (send)
-2013-11-05 20:20:06,397-0600 INFO RequestHandler Handler(tag: 7297, REGISTER) unregistering (send)
-Shutting down service...
-Shutting down blocks
-2013-11-05 20:20:15,583-0600 INFO BlockQueueProcessor Shutting down blocks
-Shutting down block Block crush.mcs.anl.gov (1x4611686018427388.000s)
-2013-11-05 20:20:15,583-0600 INFO Block Shutting down block Block crush.mcs.anl.gov (1x4611686018427388.000s)
-Adding short shutdown watchdog: count = 1, workers = 1, now = true
-2013-11-05 20:20:15,585-0600 INFO Block Adding short shutdown watchdog: count = 1, workers = 1, now = true
-Shutting down block Block grind.mcs.anl.gov (1x4611686018427388.000s)
-2013-11-05 20:20:15,585-0600 INFO Block Shutting down block Block grind.mcs.anl.gov (1x4611686018427388.000s)
-Adding short shutdown watchdog: count = 1, workers = 1, now = true
-2013-11-05 20:20:15,585-0600 INFO Block Adding short shutdown watchdog: count = 1, workers = 1, now = true
-Node [crush] 0 shut down successfully
-2013-11-05 20:20:15,586-0600 INFO Node Node [crush] 0 shut down successfully
-Shutdown sequence completed
-2013-11-05 20:20:15,586-0600 INFO CoasterService Shutdown sequence completed
-crush.mcs.anl.gov-000000: Channel shut down
-2013-11-05 20:20:15,586-0600 INFO AbstractTCPChannel crush.mcs.anl.gov-000000: Channel shut down
-MetaChannel [context: worker-1, boundTo: TCPChannel [type: server, contact: crush.mcs.anl.gov-000000]] binding to NullChannel
-2013-11-05 20:20:15,586-0600 INFO MetaChannel MetaChannel [context: worker-1, boundTo: TCPChannel [type: server, contact: crush.mcs.anl.gov-000000]] binding to NullChannel
-Using threaded sender for TCPChannel [type: server, contact: crush.mcs.anl.gov-000000]
-2013-11-05 20:20:15,587-0600 INFO AbstractStreamCoasterChannel Using threaded sender for TCPChannel [type: server, contact: crush.mcs.anl.gov-000000]
-MetaChannel [context: worker-1, boundTo: TCPChannel [type: server, contact: crush.mcs.anl.gov-000000]] binding to NullChannel
-2013-11-05 20:20:15,587-0600 INFO MetaChannel MetaChannel [context: worker-1, boundTo: TCPChannel [type: server, contact: crush.mcs.anl.gov-000000]] binding to NullChannel
-crush.mcs.anl.gov-000000: Channel terminated
-2013-11-05 20:20:15,587-0600 INFO AbstractTCPChannel crush.mcs.anl.gov-000000: Channel terminated
-Node [grind] 0 shut down successfully
-2013-11-05 20:20:15,587-0600 INFO Node Node [grind] 0 shut down successfully
-grind.mcs.anl.gov-000000: Channel shut down
-2013-11-05 20:20:15,588-0600 INFO AbstractTCPChannel grind.mcs.anl.gov-000000: Channel shut down
-MetaChannel [context: worker-2, boundTo: TCPChannel [type: server, contact: grind.mcs.anl.gov-000000]] binding to NullChannel
-2013-11-05 20:20:15,588-0600 INFO MetaChannel MetaChannel [context: worker-2, boundTo: TCPChannel [type: server, contact: grind.mcs.anl.gov-000000]] binding to NullChannel
-MetaChannel [context: worker-2, boundTo: TCPChannel [type: server, contact: grind.mcs.anl.gov-000000]] binding to NullChannel
-2013-11-05 20:20:15,588-0600 INFO MetaChannel MetaChannel [context: worker-2, boundTo: TCPChannel [type: server, contact: grind.mcs.anl.gov-000000]] binding to NullChannel
-grind.mcs.anl.gov-000000: Channel terminated
-2013-11-05 20:20:15,588-0600 INFO AbstractTCPChannel grind.mcs.anl.gov-000000: Channel terminated
-Running /homes/yadunand/swift-trunk/cog/modules/swift/dist/swift-svn/bin/coaster-service -nosec -portfile /tmp/tmp.8oWuk6Txou -localportfile /tmp/tmp.n9OHUNSS2t -passive
-Switching log to: cps-2013-11-05_20-20-53.log
-2013-11-05 20:20:53,320-0600 WARN CoasterPersistentService Switching log to: cps-2013-11-05_20-20-53.log
-Local contacts: [http://140.221.8.81:57384]
-2013-11-05 20:20:53,346-0600 INFO Settings Local contacts: [http://140.221.8.81:57384]
-Starting... id=1105-2008530
-2013-11-05 20:20:53,359-0600 INFO BlockQueueProcessor Starting... id=1105-2008530
-Started local service: http://140.221.8.81:57384
-2013-11-05 20:20:53,359-0600 INFO CoasterService Started local service: http://140.221.8.81:57384
-Started coaster service: http://140.221.8.81:59330
-2013-11-05 20:20:53,360-0600 INFO CoasterService Started coaster service: http://140.221.8.81:59330
-Started coaster service: http://140.221.8.81:59330
-Worker connection URL: http://140.221.8.81:57384
-Running ssh yadunand at crush.mcs.anl.gov mkdir -p /homes/yadunand
-Running scp /homes/yadunand/swift-trunk/cog/modules/swift/dist/swift-svn/bin/worker.pl yadunand at crush.mcs.anl.gov:/homes/yadunand
-Running ssh yadunand at crush.mcs.anl.gov WORKER_LOGGING_LEVEL=DEBUG /homes/yadunand/worker.pl http://140.221.8.81:57384 crush.mcs.anl.gov /homes/yadunand/logs
-Running ssh yadunand at grind.mcs.anl.gov mkdir -p /homes/yadunand
-Running scp /homes/yadunand/swift-trunk/cog/modules/swift/dist/swift-svn/bin/worker.pl yadunand at grind.mcs.anl.gov:/homes/yadunand
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-05 20:20:56,215-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-Channel started: TCPChannel [type: server, contact: 140.221.8.62:45374]
-2013-11-05 20:20:56,220-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 140.221.8.62:45374]
-Received registration: blockid = crush.mcs.anl.gov, url = crush
-2013-11-05 20:20:56,225-0600 INFO LocalTCPService Received registration: blockid = crush.mcs.anl.gov, url = crush
-MetaChannel [context: worker-1, boundTo: null] binding to TCPChannel [type: server, contact: 140.221.8.62:45374]
-2013-11-05 20:20:56,235-0600 INFO MetaChannel MetaChannel [context: worker-1, boundTo: null] binding to TCPChannel [type: server, contact: 140.221.8.62:45374]
-2013-11-05 20:20:56,238-0600 DEBUG Cpu worker started: block=crush.mcs.anl.gov host=crush id=0
-2013-11-05 20:20:56,239-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-Started CPU 0:1383704456s
-2013-11-05 20:20:56,240-0600 INFO Block Started CPU 0:1383704456s
-Started worker crush.mcs.anl.gov:000000
-2013-11-05 20:20:56,240-0600 INFO Block Started worker crush.mcs.anl.gov:000000
-2013-11-05 20:20:56,240-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-Using NIO sender for TCPChannel [type: server, contact: crush.mcs.anl.gov-000000]
-2013-11-05 20:20:56,240-0600 INFO AbstractStreamCoasterChannel Using NIO sender for TCPChannel [type: server, contact: crush.mcs.anl.gov-000000]
-2013-11-05 20:20:56,241-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-Handler(tag: 1498, REGISTER) unregistering (send)
-2013-11-05 20:20:56,244-0600 INFO RequestHandler Handler(tag: 1498, REGISTER) unregistering (send)
-Running ssh yadunand at grind.mcs.anl.gov WORKER_LOGGING_LEVEL=DEBUG /homes/yadunand/worker.pl http://140.221.8.81:57384 grind.mcs.anl.gov /homes/yadunand/logs
-Adjusting buffer size to 8650752 for 2 sockets
-2013-11-05 20:20:57,075-0600 INFO TCPBufferManager Adjusting buffer size to 8650752 for 2 sockets
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-05 20:20:57,076-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-Channel started: TCPChannel [type: server, contact: 140.221.8.74:49786]
-2013-11-05 20:20:57,076-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 140.221.8.74:49786]
-Received registration: blockid = grind.mcs.anl.gov, url = grind
-2013-11-05 20:20:57,076-0600 INFO LocalTCPService Received registration: blockid = grind.mcs.anl.gov, url = grind
-MetaChannel [context: worker-2, boundTo: null] binding to TCPChannel [type: server, contact: 140.221.8.74:49786]
-2013-11-05 20:20:57,076-0600 INFO MetaChannel MetaChannel [context: worker-2, boundTo: null] binding to TCPChannel [type: server, contact: 140.221.8.74:49786]
-2013-11-05 20:20:57,077-0600 DEBUG Cpu worker started: block=grind.mcs.anl.gov host=grind id=0
-2013-11-05 20:20:57,077-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Started CPU 0:1383704457s
-2013-11-05 20:20:57,077-0600 INFO Block Started CPU 0:1383704457s
-2013-11-05 20:20:57,077-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Started worker grind.mcs.anl.gov:000000
-2013-11-05 20:20:57,077-0600 INFO Block Started worker grind.mcs.anl.gov:000000
-2013-11-05 20:20:57,077-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-Handler(tag: 9070, REGISTER) unregistering (send)
-2013-11-05 20:20:57,077-0600 INFO RequestHandler Handler(tag: 9070, REGISTER) unregistering (send)
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 20:21:32,190-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 20:21:33,524-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-05 20:21:41,681-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-Multiplexer 0 started
-2013-11-05 20:21:41,684-0600 INFO Multiplexer Multiplexer 0 started
-Multiplexer 1 started
-2013-11-05 20:21:41,684-0600 INFO Multiplexer Multiplexer 1 started
-(0) Scheduling TCPChannel [type: server, contact: 140.221.8.81:58556] for addition
-2013-11-05 20:21:41,684-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 140.221.8.81:58556] for addition
-Channel started: TCPChannel [type: server, contact: 140.221.8.81:58556]
-2013-11-05 20:21:41,685-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 140.221.8.81:58556]
-Channel id: channel-1-u612953fc-1422b371dca--8000
-2013-11-05 20:21:41,706-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u612953fc-1422b371dca--8000
-MetaChannel [context: cps-58556, boundTo: null] binding to TCPChannel [type: server, contact: 140.221.8.81:58556]
-2013-11-05 20:21:41,706-0600 INFO MetaChannel MetaChannel [context: cps-58556, boundTo: null] binding to TCPChannel [type: server, contact: 140.221.8.81:58556]
-Using threaded sender for TCPChannel [type: server, contact: 140.221.8.81:58556]
-2013-11-05 20:21:41,707-0600 INFO AbstractStreamCoasterChannel Using threaded sender for TCPChannel [type: server, contact: 140.221.8.81:58556]
-Handler(tag: 17767, CHANNELCONFIG) unregistering (send)
-2013-11-05 20:21:41,707-0600 INFO RequestHandler Handler(tag: 17767, CHANNELCONFIG) unregistering (send)
-Handler(tag: 17768, CONFIGSERVICE) unregistering (send)
-2013-11-05 20:21:41,749-0600 INFO RequestHandler Handler(tag: 17768, CONFIGSERVICE) unregistering (send)
-Handler(tag: 17769, SUBMITJOB) unregistering (send)
-2013-11-05 20:21:41,842-0600 INFO RequestHandler Handler(tag: 17769, SUBMITJOB) unregistering (send)
-Handler(tag: 17770, SUBMITJOB) unregistering (send)
-2013-11-05 20:21:41,843-0600 INFO RequestHandler Handler(tag: 17770, SUBMITJOB) unregistering (send)
-2013-11-05 20:21:41,883-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-crush.mcs.anl.gov:0 (quality: 1.0) submitting urn:job-0-1383704501835-1383704501836: /bin/bash /homes/yadunand/swift-trunk/cog/modules/provider-coaster-c-client/tests/tcl/wrapper.sh
-2013-11-05 20:21:41,884-0600 INFO Cpu crush.mcs.anl.gov:0 (quality: 1.0) submitting urn:job-0-1383704501835-1383704501836: /bin/bash /homes/yadunand/swift-trunk/cog/modules/provider-coaster-c-client/tests/tcl/wrapper.sh
-2013-11-05 20:21:41,929-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 1.0) submitting urn:job-1-1383704501838-1383704501839: /bin/bash /homes/yadunand/swift-trunk/cog/modules/provider-coaster-c-client/tests/tcl/wrapper.sh
-2013-11-05 20:21:41,929-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 1.0) submitting urn:job-1-1383704501838-1383704501839: /bin/bash /homes/yadunand/swift-trunk/cog/modules/provider-coaster-c-client/tests/tcl/wrapper.sh
-LOG: /homes/yadunand/logs/worker-crush.mcs.anl.gov.log
-LOG: /homes/yadunand/logs/worker-grind.mcs.anl.gov.log
-2013-11-05 20:21:41,937-0600 DEBUG Cpu Active workerid=crush.mcs.anl.gov:000000
-Handler(tag: 1499, JOBSTATUS) unregistering (send)
-2013-11-05 20:21:41,939-0600 INFO RequestHandler Handler(tag: 1499, JOBSTATUS) unregistering (send)
-2013-11-05 20:21:41,939-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 9071, JOBSTATUS) unregistering (send)
-2013-11-05 20:21:41,939-0600 INFO RequestHandler Handler(tag: 9071, JOBSTATUS) unregistering (send)
-2013-11-05 20:21:46,980-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-05 20:21:46,980-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-05 20:21:46,980-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Handler(tag: 9072, JOBSTATUS) unregistering (send)
-2013-11-05 20:21:46,980-0600 INFO RequestHandler Handler(tag: 9072, JOBSTATUS) unregistering (send)
-2013-11-05 20:21:46,981-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-05 20:21:46,981-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-2013-11-05 20:21:46,983-0600 DEBUG Cpu Completed
-crush.mcs.anl.gov:0 jobTerminated
-2013-11-05 20:21:46,983-0600 INFO Cpu crush.mcs.anl.gov:0 jobTerminated
-2013-11-05 20:21:46,984-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-Handler(tag: 1500, JOBSTATUS) unregistering (send)
-2013-11-05 20:21:46,984-0600 INFO RequestHandler Handler(tag: 1500, JOBSTATUS) unregistering (send)
-2013-11-05 20:21:46,984-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 2
-2013-11-05 20:21:46,984-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-Avg stream buf: 0
-2013-11-05 20:21:51,695-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-05 20:22:01,707-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:22:05,847-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:22:07,798-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-05 20:22:11,721-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-05 20:22:21,734-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-05 20:22:31,748-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 20:22:38,170-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-05 20:22:41,760-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:22:43,408-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-05 20:22:51,772-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-05 20:23:01,784-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 20:23:10,493-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-05 20:23:11,796-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:23:19,018-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-05 20:23:21,807-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-05 20:23:31,818-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-05 20:23:41,829-0600 INFO Multiplexer Avg stream buf: 0
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131105-202141.842, now=131105-202342.681, channel=TCPChannel [type: server, contact: 140.221.8.81:58556]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-05 20:23:42,684-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131105-202141.842, now=131105-202342.681, channel=TCPChannel [type: server, contact: 140.221.8.81:58556]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, RLOG)
-2013-11-05 20:23:42,687-0600 INFO ChannelContext => Command(tag: 1, RLOG)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-05 20:23:42,687-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131105-202141.842, now=131105-202342.681, channel=TCPChannel [type: server, contact: 140.221.8.81:58556]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-05 20:23:42,687-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131105-202141.842, now=131105-202342.681, channel=TCPChannel [type: server, contact: 140.221.8.81:58556]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-05 20:23:42,687-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131105-202141.842, now=131105-202342.681, channel=TCPChannel [type: server, contact: 140.221.8.81:58556]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-05 20:23:42,687-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131105-202141.842, now=131105-202342.681, channel=TCPChannel [type: server, contact: 140.221.8.81:58556]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-05 20:23:42,687-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131105-202141.842, now=131105-202342.681, channel=TCPChannel [type: server, contact: 140.221.8.81:58556]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-05 20:23:42,688-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131105-202141.842, now=131105-202342.681, channel=TCPChannel [type: server, contact: 140.221.8.81:58556]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 5, JOBSTATUS)
-2013-11-05 20:23:42,688-0600 INFO ChannelContext => Command(tag: 5, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131105-202141.842, now=131105-202342.681, channel=TCPChannel [type: server, contact: 140.221.8.81:58556]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-05 20:23:42,688-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131105-202141.842, now=131105-202342.681, channel=TCPChannel [type: server, contact: 140.221.8.81:58556]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 6, JOBSTATUS)
-2013-11-05 20:23:42,688-0600 INFO ChannelContext => Command(tag: 6, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131105-202141.842, now=131105-202342.681, channel=TCPChannel [type: server, contact: 140.221.8.81:58556]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-05 20:23:42,688-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131105-202141.842, now=131105-202342.681, channel=TCPChannel [type: server, contact: 140.221.8.81:58556]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 7, JOBSTATUS)
-2013-11-05 20:23:42,689-0600 INFO ChannelContext => Command(tag: 7, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131105-202141.842, now=131105-202342.681, channel=TCPChannel [type: server, contact: 140.221.8.81:58556]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-05 20:23:42,689-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131105-202141.842, now=131105-202342.681, channel=TCPChannel [type: server, contact: 140.221.8.81:58556]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 8, JOBSTATUS)
-2013-11-05 20:23:42,689-0600 INFO ChannelContext => Command(tag: 8, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131105-202141.842, now=131105-202342.681, channel=TCPChannel [type: server, contact: 140.221.8.81:58556]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-05 20:23:42,689-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131105-202141.842, now=131105-202342.681, channel=TCPChannel [type: server, contact: 140.221.8.81:58556]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 9, JOBSTATUS)
-2013-11-05 20:23:42,689-0600 INFO ChannelContext => Command(tag: 9, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131105-202141.842, now=131105-202342.681, channel=TCPChannel [type: server, contact: 140.221.8.81:58556]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-05 20:23:42,689-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131105-202141.842, now=131105-202342.681, channel=TCPChannel [type: server, contact: 140.221.8.81:58556]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-05 20:23:42,690-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 140.221.8.81:58556] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131105-202141.842, now=131105-202342.681, channel=TCPChannel [type: server, contact: 140.221.8.81:58556]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-05 20:23:42,690-0600 INFO ChannelManager TCPChannel [type: server, contact: 140.221.8.81:58556] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131105-202141.842, now=131105-202342.681, channel=TCPChannel [type: server, contact: 140.221.8.81:58556]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-null: Channel shut down
-2013-11-05 20:23:42,690-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-58556, boundTo: TCPChannel [type: server, contact: 140.221.8.81:58556]] binding to NullChannel
-2013-11-05 20:23:42,691-0600 INFO MetaChannel MetaChannel [context: cps-58556, boundTo: TCPChannel [type: server, contact: 140.221.8.81:58556]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 140.221.8.81:58556]
-2013-11-05 20:23:42,691-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 140.221.8.81:58556]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131105-202141.842, now=131105-202342.681, channel=TCPChannel [type: server, contact: 140.221.8.81:58556]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-05 20:23:42,691-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131105-202141.842, now=131105-202342.681, channel=TCPChannel [type: server, contact: 140.221.8.81:58556]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, RLOG)
-2013-11-05 20:23:42,691-0600 INFO ChannelContext => Command(tag: 1, RLOG)
-Channel is closed. Removing.
-2013-11-05 20:23:42,691-0600 INFO Multiplexer Channel is closed. Removing.
-=> Command(tag: 2, JOBSTATUS)
-2013-11-05 20:23:42,692-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-05 20:23:42,692-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131105-202141.842, now=131105-202342.681, channel=TCPChannel [type: server, contact: 140.221.8.81:58556]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-05 20:23:42,692-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131105-202141.842, now=131105-202342.681, channel=TCPChannel [type: server, contact: 140.221.8.81:58556]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-05 20:23:42,692-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131105-202141.842, now=131105-202342.681, channel=TCPChannel [type: server, contact: 140.221.8.81:58556]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-05 20:23:42,692-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131105-202141.842, now=131105-202342.681, channel=TCPChannel [type: server, contact: 140.221.8.81:58556]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-05 20:23:42,693-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131105-202141.842, now=131105-202342.681, channel=TCPChannel [type: server, contact: 140.221.8.81:58556]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-05 20:23:42,693-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131105-202141.842, now=131105-202342.681, channel=TCPChannel [type: server, contact: 140.221.8.81:58556]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 5, JOBSTATUS)
-2013-11-05 20:23:42,693-0600 INFO ChannelContext => Command(tag: 5, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131105-202141.842, now=131105-202342.681, channel=TCPChannel [type: server, contact: 140.221.8.81:58556]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-05 20:23:42,693-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131105-202141.842, now=131105-202342.681, channel=TCPChannel [type: server, contact: 140.221.8.81:58556]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 6, JOBSTATUS)
-2013-11-05 20:23:42,693-0600 INFO ChannelContext => Command(tag: 6, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131105-202141.842, now=131105-202342.681, channel=TCPChannel [type: server, contact: 140.221.8.81:58556]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-05 20:23:42,693-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131105-202141.842, now=131105-202342.681, channel=TCPChannel [type: server, contact: 140.221.8.81:58556]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 7, JOBSTATUS)
-2013-11-05 20:23:42,694-0600 INFO ChannelContext => Command(tag: 7, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131105-202141.842, now=131105-202342.681, channel=TCPChannel [type: server, contact: 140.221.8.81:58556]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-05 20:23:42,694-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131105-202141.842, now=131105-202342.681, channel=TCPChannel [type: server, contact: 140.221.8.81:58556]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 8, JOBSTATUS)
-2013-11-05 20:23:42,694-0600 INFO ChannelContext => Command(tag: 8, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131105-202141.842, now=131105-202342.681, channel=TCPChannel [type: server, contact: 140.221.8.81:58556]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-05 20:23:42,694-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131105-202141.842, now=131105-202342.681, channel=TCPChannel [type: server, contact: 140.221.8.81:58556]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 9, JOBSTATUS)
-2013-11-05 20:23:42,694-0600 INFO ChannelContext => Command(tag: 9, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131105-202141.842, now=131105-202342.681, channel=TCPChannel [type: server, contact: 140.221.8.81:58556]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-05 20:23:42,694-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131105-202141.842, now=131105-202342.681, channel=TCPChannel [type: server, contact: 140.221.8.81:58556]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel exception handled
-2013-11-05 20:23:42,694-0600 INFO ChannelManager Channel exception handled
-Channel config: null -> {keepalive=null}
-2013-11-05 20:23:42,695-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 140.221.8.81:58556] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-05 20:23:42,695-0600 INFO ChannelManager TCPChannel [type: server, contact: 140.221.8.81:58556] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-05 20:23:42,695-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 140.221.8.81:58556]
-2013-11-05 20:23:42,695-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 140.221.8.81:58556]
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 20:23:42,816-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 20:23:51,842-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:23:54,628-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 20:24:01,850-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:24:11,861-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 20:24:15,140-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 20:24:21,870-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 20:24:30,238-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 20:24:31,878-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:24:41,885-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 20:24:47,463-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 20:24:51,893-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:25:01,901-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:25:05,848-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 20:25:11,909-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 20:25:19,786-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 20:25:21,916-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:25:31,924-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:25:41,458-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 20:25:41,931-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:25:51,939-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 20:25:52,109-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 20:26:01,946-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:26:11,954-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 20:26:17,068-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 20:26:21,961-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 20:26:24,431-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 20:26:31,969-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:26:41,976-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:26:51,983-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:26:52,678-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:26:56,755-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 20:27:01,990-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:27:11,997-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:27:22,005-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 20:27:28,287-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 20:27:29,078-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 20:27:32,012-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:27:42,020-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:27:52,028-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 20:28:01,401-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 20:28:02,036-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 20:28:03,898-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 20:28:12,044-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:28:22,052-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:28:32,059-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 20:28:33,724-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:28:39,508-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 20:28:42,067-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:28:52,075-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:29:02,083-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 20:29:06,047-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 20:29:12,091-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 20:29:15,117-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 20:29:22,098-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:29:32,106-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:29:38,370-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 20:29:42,114-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:29:50,728-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 20:29:52,122-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:30:02,130-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:30:10,693-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 20:30:12,138-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:30:22,145-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:30:26,338-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 20:30:32,153-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:30:42,161-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 20:30:43,016-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 20:30:52,169-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 20:31:01,947-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 20:31:02,177-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:31:12,184-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 20:31:15,339-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 20:31:22,192-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:31:32,200-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:31:37,558-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 20:31:42,208-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 20:31:47,661-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 20:31:52,216-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:32:02,223-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:32:12,231-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:32:13,168-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 20:32:19,985-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 20:32:22,239-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:32:32,247-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:32:42,255-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:32:48,778-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 20:32:52,262-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:32:52,308-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 20:33:02,270-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:33:12,278-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:33:22,286-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:33:24,388-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 20:33:24,631-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 20:33:32,293-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:33:42,301-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:33:52,309-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 20:33:56,954-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 20:33:59,997-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 20:34:02,317-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:34:12,325-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:34:22,333-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-05 20:34:29,278-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-05 20:34:32,340-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 20:34:35,608-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 20:34:42,348-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:34:52,356-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 20:35:01,600-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 20:35:02,364-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:35:11,218-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 20:35:12,371-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:35:22,379-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:35:32,387-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 20:35:33,923-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 20:35:42,395-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:35:46,828-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 20:35:52,402-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:36:02,410-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 20:36:06,246-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 20:36:12,418-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:36:22,425-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 20:36:22,438-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 20:36:32,433-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:36:38,570-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 20:36:42,441-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:36:52,448-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 20:36:58,048-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 20:37:02,456-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 20:37:10,893-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 20:37:12,463-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:37:22,471-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:37:32,479-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:37:33,658-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 20:37:42,487-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 20:37:43,216-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 20:37:52,494-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:38:02,502-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 20:38:09,267-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 20:38:12,510-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 20:38:15,539-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 20:38:22,518-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:38:32,526-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:38:42,534-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:38:44,878-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 20:38:47,862-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 20:38:52,541-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:39:02,549-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:39:12,557-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 20:39:20,185-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:39:20,488-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 20:39:22,565-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:39:32,572-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:39:42,580-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 20:39:52,508-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 20:39:52,588-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:39:56,098-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 20:40:02,596-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:40:12,604-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:40:22,611-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 20:40:24,831-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 20:40:31,708-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 20:40:32,619-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:40:42,627-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:40:52,635-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:40:57,154-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 20:41:02,642-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:41:07,318-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 20:41:12,650-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:41:22,658-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 20:41:29,477-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 20:41:32,666-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:41:42,673-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:41:42,928-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 20:41:52,681-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 20:42:01,800-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 20:42:02,689-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:42:12,697-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:42:18,538-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 20:42:22,704-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:42:32,712-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:42:34,123-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 20:42:42,720-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:42:52,728-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:42:54,148-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 20:43:02,735-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:43:06,446-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 20:43:12,743-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:43:22,751-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:43:29,758-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 20:43:32,759-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 20:43:38,769-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 20:43:42,766-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:43:52,774-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:44:02,782-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:44:05,368-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 20:44:11,092-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 20:44:12,789-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:44:22,797-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:44:32,805-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:44:40,978-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 20:44:42,813-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 20:44:43,415-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 20:44:52,820-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:45:02,829-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:45:12,837-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 20:45:15,738-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:45:16,588-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 20:45:22,845-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:45:32,853-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:45:42,860-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 20:45:48,061-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:45:52,198-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 20:45:52,868-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:46:02,876-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:46:12,884-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 20:46:20,384-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 20:46:22,892-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 20:46:27,808-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 20:46:32,899-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:46:42,907-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 20:46:52,706-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 20:46:52,915-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:47:02,923-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:47:03,418-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 20:47:12,931-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:47:22,938-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:47:25,030-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 20:47:32,946-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 20:47:39,027-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 20:47:42,954-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:47:52,962-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 20:47:57,353-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 20:48:02,970-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:48:12,977-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:48:14,638-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 20:48:22,985-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 20:48:29,676-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 20:48:32,993-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:48:43,001-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:48:50,248-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 20:48:53,008-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 20:49:01,999-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 20:49:03,017-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:49:13,025-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:49:23,034-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:49:25,858-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 20:49:33,042-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 20:49:34,322-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 20:49:43,050-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:49:53,058-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:50:01,468-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 20:50:03,067-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 20:50:06,645-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 20:50:13,075-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:50:23,082-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:50:33,090-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:50:37,078-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 20:50:38,968-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 20:50:43,098-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:50:53,106-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:51:03,113-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 20:51:11,291-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 20:51:12,687-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 20:51:13,121-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:51:23,128-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:51:33,136-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:51:43,144-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 20:51:43,613-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:51:48,298-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 20:51:53,151-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:52:03,159-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:52:13,167-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:52:15,937-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 20:52:23,174-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:52:23,908-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 20:52:33,182-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:52:43,190-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 20:52:48,260-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 20:52:53,198-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 20:52:59,518-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 20:53:03,206-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:53:13,214-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-05 20:53:20,584-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-05 20:53:23,222-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:53:33,230-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:53:35,129-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 20:53:43,238-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 20:53:52,906-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 20:53:53,246-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:54:03,254-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:54:10,739-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 20:54:13,262-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:54:23,269-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:54:25,229-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 20:54:33,277-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:54:43,285-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 20:54:46,348-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 20:54:53,293-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 20:54:57,552-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 20:55:03,301-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:55:13,309-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:55:21,959-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 20:55:23,316-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 20:55:29,875-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 20:55:33,324-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:55:43,332-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:55:53,340-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:55:57,569-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 20:56:02,198-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 20:56:03,348-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:56:13,356-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:56:23,364-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 20:56:33,179-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 20:56:33,372-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:56:34,522-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 20:56:43,380-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:56:53,388-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:57:03,396-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:57:06,845-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:57:08,789-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 20:57:13,404-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:57:23,411-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:57:33,419-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 20:57:39,168-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 20:57:43,427-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:57:44,399-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 20:57:53,435-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:58:03,443-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 20:58:11,491-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 20:58:13,451-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:58:20,009-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 20:58:23,459-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:58:33,467-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:58:43,475-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 20:58:43,814-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 20:58:53,484-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:58:55,619-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 20:59:03,492-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:59:13,500-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 20:59:16,137-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 20:59:23,508-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 20:59:31,229-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 20:59:33,516-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 20:59:43,524-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 20:59:48,461-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 20:59:53,533-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:00:03,541-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:00:06,840-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:00:13,549-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:00:20,784-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:00:23,557-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:00:33,566-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:00:42,450-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:00:43,574-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:00:53,107-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:00:53,582-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:01:03,590-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:01:13,598-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:01:18,060-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:01:23,607-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:01:25,430-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:01:33,615-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:01:43,623-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:01:53,631-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:01:53,670-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:01:57,753-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:02:03,639-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:02:13,647-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:02:23,656-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 21:02:29,280-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 21:02:30,075-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 21:02:33,664-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:02:43,672-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:02:53,680-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:03:02,399-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:03:03,688-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:03:04,891-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:03:13,696-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:03:23,704-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:03:33,713-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:03:34,722-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:03:40,501-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:03:43,721-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:03:53,729-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:04:03,737-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:04:07,045-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:04:13,745-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:04:16,111-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:04:23,753-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:04:33,761-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:04:39,368-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:04:43,769-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:04:51,721-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:04:53,778-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:05:03,786-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:05:11,691-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:05:13,794-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:05:23,802-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:05:27,331-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:05:33,810-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:05:43,818-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:05:44,014-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:05:53,826-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:06:02,941-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:06:03,835-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:06:13,843-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:06:16,337-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:06:23,851-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:06:33,859-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:06:38,551-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:06:43,867-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:06:48,660-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:06:53,875-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:07:03,883-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:07:13,891-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:07:14,161-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:07:20,983-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:07:23,899-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:07:33,907-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:07:43,915-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:07:49,771-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 21:07:53,305-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 21:07:53,923-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:08:03,931-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:08:13,939-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:08:23,947-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:08:25,381-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:08:25,629-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:08:33,955-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:08:43,963-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:08:53,971-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:08:57,952-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:09:00,991-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:09:03,980-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:09:13,988-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:09:23,996-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:09:30,275-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:09:34,004-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:09:36,601-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:09:44,012-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:09:54,020-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:10:02,598-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:10:04,028-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:10:12,211-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:10:14,036-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:10:24,044-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:10:34,052-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:10:34,921-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:10:44,060-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:10:47,821-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:10:54,068-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:11:04,076-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 21:11:07,243-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 21:11:14,084-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:11:23,431-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:11:24,092-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:11:34,100-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:11:39,567-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:11:44,108-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:11:54,116-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:11:59,041-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:12:04,124-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:12:11,890-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:12:14,132-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:12:24,140-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:12:34,148-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:12:34,651-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:12:44,157-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:12:44,213-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:12:54,164-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:13:04,172-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:13:10,261-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:13:14,180-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:13:16,536-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:13:24,188-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:13:34,197-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:13:44,205-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:13:45,871-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:13:48,859-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:13:54,213-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:14:04,221-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:14:14,229-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:14:21,182-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:14:21,481-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:14:24,237-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:14:34,246-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:14:44,254-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:14:53,505-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:14:54,262-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:14:57,091-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:15:04,270-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:15:14,278-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:15:24,287-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-05 21:15:25,829-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:15:32,701-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:15:34,295-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:15:44,303-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:15:54,312-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:15:58,151-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:16:04,320-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:16:08,311-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:16:14,327-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:16:24,335-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:16:30,474-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:16:34,343-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 21:16:43,921-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 21:16:44,351-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:16:54,359-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:17:02,797-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:17:04,366-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:17:14,374-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:17:19,532-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:17:24,382-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:17:34,390-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:17:35,120-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:17:44,398-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:17:54,405-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 21:17:55,141-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 21:18:04,413-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:18:07,443-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:18:14,421-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:18:24,428-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:18:30,752-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:18:34,436-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:18:39,766-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:18:44,444-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:18:54,452-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:19:04,459-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:19:06,362-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:19:12,089-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:19:14,467-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:19:24,475-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:19:34,483-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:19:41,972-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 21:19:44,412-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 21:19:44,490-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:19:54,498-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:20:04,506-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:20:14,514-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:20:16,736-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:20:17,582-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:20:24,522-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:20:34,530-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:20:44,538-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:20:49,060-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:20:53,192-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:20:54,546-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:21:04,553-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:21:14,561-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:21:21,383-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:21:24,569-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 21:21:28,801-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 21:21:34,577-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:21:44,585-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:21:53,706-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:21:54,593-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:22:04,412-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:22:04,601-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:22:14,609-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:22:24,617-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:22:26,029-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:22:34,625-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 21:22:40,021-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 21:22:44,633-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:22:54,641-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:22:58,352-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:23:04,649-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:23:14,657-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:23:15,632-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:23:24,664-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:23:30,675-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:23:34,672-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:23:44,680-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 21:23:51,241-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 21:23:54,688-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:24:02,998-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:24:04,696-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:24:14,704-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:24:24,712-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:24:26,852-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:24:34,720-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:24:35,321-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:24:44,728-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:24:54,736-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:25:02,462-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:25:04,744-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:25:07,644-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:25:14,752-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:25:24,760-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:25:34,768-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:25:38,073-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:25:39,967-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:25:44,776-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:25:54,784-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:26:04,792-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:26:12,290-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:26:13,683-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:26:14,800-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:26:24,808-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:26:34,816-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:26:44,613-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:26:44,824-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 21:26:49,293-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 21:26:54,833-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:27:04,841-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:27:14,849-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:27:16,936-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:27:24,857-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:27:24,904-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:27:34,865-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:27:44,873-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:27:49,259-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:27:54,881-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:28:00,514-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:28:04,890-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:28:14,898-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:28:21,582-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:28:24,906-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:28:34,914-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 21:28:36,125-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 21:28:44,922-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 21:28:53,904-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 21:28:54,931-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:29:04,939-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:29:11,735-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:29:14,947-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:29:24,955-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:29:26,228-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:29:34,963-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:29:44,972-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:29:47,345-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:29:54,980-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 21:29:58,550-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 21:30:04,988-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:30:14,996-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:30:22,956-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:30:25,004-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:30:30,874-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:30:35,012-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:30:45,020-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:30:55,028-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:30:58,566-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:31:03,197-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:31:05,037-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:31:15,045-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:31:25,053-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:31:34,176-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:31:35,061-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 21:31:35,519-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 21:31:45,069-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:31:55,077-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:32:05,085-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 21:32:07,843-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:32:09,786-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:32:15,093-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:32:25,101-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:32:35,110-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:32:40,166-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:32:45,118-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:32:45,396-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:32:55,126-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:33:05,134-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:33:12,489-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:33:15,142-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 21:33:21,006-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 21:33:25,150-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:33:35,158-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:33:44,812-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:33:45,166-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:33:55,174-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:33:56,617-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:34:05,182-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:34:15,190-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:34:17,135-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:34:25,198-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:34:32,227-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:34:35,206-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:34:45,215-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:34:49,458-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:34:55,223-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:35:05,231-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:35:07,837-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:35:15,239-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:35:21,781-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:35:25,247-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:35:35,255-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:35:43,447-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:35:45,263-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:35:54,104-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:35:55,271-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:36:05,279-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:36:15,287-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:36:19,057-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:36:25,295-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:36:26,427-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:36:35,303-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:36:45,311-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:36:54,667-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:36:55,319-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:36:58,750-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:37:05,327-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:37:15,335-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:37:25,343-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:37:30,277-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:37:31,073-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:37:35,352-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:37:45,360-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:37:55,368-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:38:03,396-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:38:05,376-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 21:38:05,887-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 21:38:15,384-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:38:25,392-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:38:35,400-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:38:35,719-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:38:41,498-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:38:45,408-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:38:55,416-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:39:05,424-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:39:08,042-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:39:15,432-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:39:17,108-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:39:25,440-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:39:35,449-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:39:40,365-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:39:45,457-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:39:52,718-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:39:55,465-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:40:05,473-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:40:12,688-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:40:15,481-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:40:25,489-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:40:28,328-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:40:35,498-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:40:45,011-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:40:45,506-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:40:55,514-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:41:03,938-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:41:05,522-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:41:15,530-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:41:17,334-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:41:25,538-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:41:35,546-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:41:39,548-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:41:45,554-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:41:49,657-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:41:55,563-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:42:05,571-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:42:15,158-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:42:15,579-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:42:21,980-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:42:25,587-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:42:35,596-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:42:45,604-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:42:50,768-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:42:54,303-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:42:55,612-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:43:05,621-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:43:15,629-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:43:25,637-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:43:26,378-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:43:26,626-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:43:35,646-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:43:45,654-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:43:55,662-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:43:58,949-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:44:01,988-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:44:05,671-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:44:15,679-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:44:25,687-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:44:31,272-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:44:35,695-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:44:37,598-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:44:45,703-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:44:55,711-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:45:03,595-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:45:05,720-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:45:13,208-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:45:15,728-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:45:25,736-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:45:35,744-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:45:35,918-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:45:45,753-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 21:45:48,818-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 21:45:55,761-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:46:05,769-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:46:08,241-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:46:15,777-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:46:24,429-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:46:25,785-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:46:35,794-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:46:40,564-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:46:45,802-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:46:55,810-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:47:00,039-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:47:05,818-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:47:12,887-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:47:15,826-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:47:25,834-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:47:35,649-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:47:35,843-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:47:45,210-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:47:45,851-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:47:55,859-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:48:05,867-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:48:11,259-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:48:15,875-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:48:17,533-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:48:25,883-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:48:35,892-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:48:45,900-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:48:46,869-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:48:49,856-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:48:55,908-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:49:05,916-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:49:15,925-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:49:22,179-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:49:22,479-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:49:25,933-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:49:35,941-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:49:45,949-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:49:54,502-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:49:55,957-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:49:58,089-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:50:05,965-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:50:15,974-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:50:25,982-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:50:26,825-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:50:33,699-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:50:35,990-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:50:45,998-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:50:56,006-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:50:59,148-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:51:06,014-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:51:09,309-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:51:16,022-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:51:26,029-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:51:31,471-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:51:36,037-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 21:51:44,920-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 21:51:46,044-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:51:56,051-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:52:03,794-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:52:06,058-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:52:16,065-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:52:20,530-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:52:26,073-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:52:36,082-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:52:36,117-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:52:46,090-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:52:56,098-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:52:56,140-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:53:06,106-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:53:08,440-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:53:16,114-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:53:26,122-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:53:31,750-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:53:36,130-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:53:40,763-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:53:46,139-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:53:56,147-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:54:06,155-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:54:07,360-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:54:13,086-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:54:16,162-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:54:26,170-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:54:36,178-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:54:42,970-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:54:45,409-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:54:46,186-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:54:56,194-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:55:06,202-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:55:16,209-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:55:17,732-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:55:18,580-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:55:26,217-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:55:36,225-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:55:46,233-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:55:50,055-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:55:54,190-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:55:56,241-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:56:06,249-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:56:16,257-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 21:56:22,377-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 21:56:26,264-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 21:56:29,800-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 21:56:36,272-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:56:46,280-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 21:56:54,700-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 21:56:56,288-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:57:05,411-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:57:06,296-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:57:16,304-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:57:26,312-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:57:27,024-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:57:36,319-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:57:41,022-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 21:57:46,327-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:57:56,335-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:57:59,347-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:58:06,343-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:58:16,351-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:58:16,632-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:58:26,359-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:58:31,670-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:58:36,367-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:58:46,375-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 21:58:52,241-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 21:58:56,383-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:59:03,993-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:59:06,391-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:59:16,399-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:59:26,407-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 21:59:27,852-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 21:59:36,316-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 21:59:36,415-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:59:46,423-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 21:59:56,431-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:00:03,462-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:00:06,439-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:00:08,639-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:00:16,446-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:00:26,454-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:00:36,462-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:00:39,072-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:00:40,962-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:00:46,470-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:00:56,479-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:01:06,486-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:01:13,285-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 22:01:14,682-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 22:01:16,494-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:01:26,502-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:01:36,510-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:01:45,608-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:01:46,519-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 22:01:50,293-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 22:01:56,527-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:02:06,535-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:02:16,543-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:02:17,931-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:02:25,903-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:02:26,551-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:02:36,559-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:02:46,567-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:02:50,254-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:02:56,575-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 22:03:01,513-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 22:03:06,582-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:03:16,590-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:03:22,577-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:03:26,598-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:03:36,606-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:03:37,123-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:03:46,614-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:03:54,900-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:03:56,622-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:04:06,630-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 22:04:12,732-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 22:04:16,638-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:04:26,646-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 22:04:27,223-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 22:04:36,654-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:04:46,662-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:04:48,343-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:04:56,669-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 22:04:59,547-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 22:05:06,677-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:05:16,685-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 22:05:23,954-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 22:05:26,693-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 22:05:31,871-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 22:05:36,701-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:05:46,709-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:05:56,717-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:05:59,563-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 22:06:04,194-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 22:06:06,725-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:06:16,733-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:06:26,741-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 22:06:35,173-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:06:36,517-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:06:36,749-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:06:46,757-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:06:56,765-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:07:06,772-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 22:07:08,839-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:07:10,784-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:07:16,780-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:07:26,788-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:07:36,796-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:07:41,163-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 22:07:46,394-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 22:07:46,804-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:07:56,812-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:08:06,820-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:08:13,486-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:08:16,828-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 22:08:22,005-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 22:08:26,836-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:08:36,844-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:08:45,809-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:08:46,852-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:08:56,860-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:08:57,615-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:09:06,868-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:09:16,876-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:09:18,132-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:09:26,884-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 22:09:33,225-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 22:09:36,892-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:09:46,900-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:09:50,455-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:09:56,908-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:10:06,916-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:10:08,835-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:10:16,924-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:10:22,778-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:10:26,933-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:10:36,941-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 22:10:44,445-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 22:10:46,949-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:10:55,101-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:10:56,957-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:11:06,966-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:11:16,974-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:11:20,055-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:11:26,982-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:11:27,424-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:11:36,990-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:11:46,999-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:11:55,665-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:11:57,007-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:11:59,747-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:12:07,015-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:12:17,024-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:12:27,032-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 22:12:31,275-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:12:32,070-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:12:37,040-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:12:47,048-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:12:57,056-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:13:04,393-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:13:06,885-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:13:07,065-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:13:17,073-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:13:27,081-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:13:36,716-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:13:37,089-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 22:13:42,495-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 22:13:47,098-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:13:57,106-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:14:07,114-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:14:09,039-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:14:17,122-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:14:18,105-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:14:27,130-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:14:37,138-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:14:41,362-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:14:47,146-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:14:53,715-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:14:57,155-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:15:07,163-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:15:13,685-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:15:17,171-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:15:27,179-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:15:29,325-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:15:37,187-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:15:46,008-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:15:47,195-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:15:57,203-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:16:04,935-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:16:07,212-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:16:17,220-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:16:18,331-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:16:27,228-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:16:37,236-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 22:16:40,545-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 22:16:47,244-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:16:50,654-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:16:57,252-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:17:07,260-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 22:17:16,155-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 22:17:17,268-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:17:22,977-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:17:27,276-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:17:37,284-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:17:47,292-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:17:51,765-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:17:55,300-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:17:57,300-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:18:07,308-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:18:17,316-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:18:27,324-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 22:18:27,375-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:18:27,623-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:18:37,332-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:18:47,340-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:18:57,348-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:18:59,946-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:19:02,985-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:19:07,356-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:19:17,365-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:19:27,373-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:19:32,269-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:19:37,381-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 22:19:38,595-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 22:19:47,389-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:19:57,397-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:20:04,592-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:20:07,406-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:20:14,205-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:20:17,414-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:20:27,422-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:20:36,915-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:20:37,430-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:20:47,439-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 22:20:49,815-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 22:20:57,447-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:21:07,455-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:21:09,238-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:21:17,463-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:21:25,425-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:21:27,471-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:21:37,480-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:21:41,561-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:21:47,488-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:21:57,496-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:22:01,035-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:22:07,504-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:22:13,884-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:22:17,512-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:22:27,520-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 22:22:36,645-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 22:22:37,528-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:22:46,207-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:22:47,536-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:22:57,544-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:23:07,552-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:23:12,255-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:23:17,560-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:23:18,530-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:23:27,568-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:23:37,575-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:23:47,583-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:23:47,865-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:23:50,853-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:23:57,591-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:24:07,599-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:24:17,607-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:24:23,176-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 22:24:23,475-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 22:24:27,615-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:24:37,623-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:24:47,631-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:24:55,499-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:24:57,639-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:24:59,085-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:25:07,647-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:25:17,655-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:25:27,663-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:25:27,822-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 22:25:34,695-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 22:25:37,671-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:25:47,679-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:25:57,687-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:26:00,145-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:26:07,695-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:26:10,305-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:26:17,703-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:26:27,711-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:26:32,468-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:26:37,718-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:26:45,915-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:26:47,726-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:26:57,734-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 22:27:04,791-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 22:27:07,742-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:27:17,750-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 22:27:21,525-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 22:27:27,758-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:27:37,114-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:27:37,766-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:27:47,774-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 22:27:57,135-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 22:27:57,782-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:28:07,790-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 22:28:09,437-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 22:28:17,797-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:28:27,805-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-05 22:28:32,746-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-05 22:28:37,813-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 22:28:41,759-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 22:28:47,821-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:28:57,829-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:29:07,837-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 22:29:08,355-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 22:29:14,082-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 22:29:17,845-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:29:27,853-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:29:37,861-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:29:43,965-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 22:29:46,405-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 22:29:47,868-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:29:57,876-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:30:07,884-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:30:17,892-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 22:30:18,729-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 22:30:19,575-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 22:30:27,900-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:30:37,908-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:30:47,916-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 22:30:51,053-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:30:55,185-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:30:57,923-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:31:07,931-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:31:17,939-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 22:31:23,374-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 22:31:27,947-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 22:31:30,795-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 22:31:37,955-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:31:47,963-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:31:55,698-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:31:57,971-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:32:06,405-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:32:07,979-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:32:17,987-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:32:27,995-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:32:28,021-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:32:38,003-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 22:32:42,016-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 22:32:48,011-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:32:58,019-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:33:00,344-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:33:08,027-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 22:33:17,626-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 22:33:18,035-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:33:28,043-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:33:32,667-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:33:38,052-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:33:48,060-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:33:53,236-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:33:58,068-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 22:34:04,990-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 22:34:08,076-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:34:18,084-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:34:28,092-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 22:34:28,846-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:34:37,313-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:34:38,100-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:34:48,108-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:34:58,116-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 22:35:04,456-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 22:35:08,124-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:35:09,636-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:35:18,132-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:35:28,140-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:35:38,148-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-05 22:35:40,067-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:35:41,959-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:35:48,156-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:35:58,164-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:36:08,172-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:36:14,282-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 22:36:15,676-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 22:36:18,180-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:36:28,188-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:36:38,196-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 22:36:46,605-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 22:36:48,204-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:36:51,286-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:36:58,212-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:37:08,220-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:37:18,228-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 22:37:18,928-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 22:37:26,896-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 22:37:28,236-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:37:38,244-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:37:48,252-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:37:51,251-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:37:58,260-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:38:02,506-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:38:08,268-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:38:18,276-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 22:38:23,574-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 22:38:28,284-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:38:38,116-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:38:38,292-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:38:48,300-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 22:38:55,896-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 22:38:58,308-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:39:08,316-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 22:39:13,726-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 22:39:18,324-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 22:39:28,220-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 22:39:28,332-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:39:38,340-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:39:48,348-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:39:49,336-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:39:58,356-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 22:40:00,542-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 22:40:08,364-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:40:18,372-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-05 22:40:24,947-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-05 22:40:28,380-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 22:40:32,866-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 22:40:38,388-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:40:48,396-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:40:58,404-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 22:41:00,556-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 22:41:05,188-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 22:41:08,412-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:41:18,420-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:41:28,428-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:41:36,166-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 22:41:37,511-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 22:41:38,436-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:41:48,443-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:41:58,451-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:42:08,459-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:42:09,835-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 22:42:11,776-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 22:42:18,467-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:42:28,475-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:42:38,483-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:42:42,158-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:42:47,386-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:42:48,491-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:42:58,499-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:43:08,507-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:43:14,481-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:43:18,515-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 22:43:22,997-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 22:43:28,523-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:43:38,531-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:43:46,804-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:43:48,539-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:43:58,547-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 22:43:58,607-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 22:44:08,555-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:44:18,563-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:44:19,127-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:44:28,571-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:44:34,217-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:44:38,579-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:44:48,587-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:44:51,450-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:44:58,595-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:45:08,603-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 22:45:09,827-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 22:45:18,611-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:45:23,773-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:45:28,619-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:45:38,627-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:45:45,437-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:45:48,635-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 22:45:56,096-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 22:45:58,643-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:46:08,651-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:46:18,659-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:46:21,047-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 22:46:28,420-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 22:46:28,667-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:46:38,675-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:46:48,683-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 22:46:56,657-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 22:46:58,684-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:47:00,743-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:47:08,692-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:47:18,700-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:47:28,708-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:47:32,267-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:47:33,066-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:47:38,716-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:47:48,724-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:47:58,732-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:48:05,389-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 22:48:07,877-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 22:48:08,740-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:48:18,748-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:48:28,756-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:48:37,712-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:48:38,764-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:48:43,487-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:48:48,772-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:48:58,780-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:49:08,788-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:49:10,035-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:49:18,796-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:49:19,097-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:49:28,804-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:49:38,816-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:49:42,358-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:49:48,824-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 22:49:54,707-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 22:49:58,832-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:50:08,840-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:50:14,681-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:50:18,848-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:50:28,856-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:50:30,317-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:50:38,864-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:50:47,004-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:50:48,872-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:50:58,880-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 22:51:05,927-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 22:51:08,887-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:51:18,895-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:51:19,327-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:51:28,903-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:51:38,911-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 22:51:41,537-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 22:51:48,919-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:51:51,650-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:51:58,927-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:52:08,935-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:52:17,147-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:52:18,943-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:52:23,973-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:52:28,951-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:52:38,959-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:52:48,966-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 22:52:52,757-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:52:56,296-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:52:58,974-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:53:08,982-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:53:18,990-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:53:28,367-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:53:28,619-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:53:28,998-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:53:39,006-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:53:49,014-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:53:59,022-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:54:00,942-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 22:54:03,977-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 22:54:09,030-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:54:19,038-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:54:29,046-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:54:33,265-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:54:39,054-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 22:54:39,587-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 22:54:49,062-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:54:59,070-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:55:05,588-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:55:09,078-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:55:15,197-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:55:19,086-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:55:29,094-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:55:37,911-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:55:39,102-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:55:49,110-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 22:55:50,807-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 22:55:59,118-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:56:09,126-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:56:10,234-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:56:19,134-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:56:26,417-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:56:29,142-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:56:39,150-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:56:42,557-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:56:49,158-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:56:59,166-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:57:02,027-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:57:09,174-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:57:14,880-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:57:19,182-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:57:29,190-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 22:57:37,637-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 22:57:39,198-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:57:47,203-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:57:49,206-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:57:59,214-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:58:09,222-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:58:13,247-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:58:19,230-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:58:19,526-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:58:29,238-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:58:39,246-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:58:48,857-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:58:49,255-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:58:51,849-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:58:59,263-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:59:09,271-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:59:19,279-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:59:24,172-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:59:24,467-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:59:29,288-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:59:39,296-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 22:59:49,304-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 22:59:56,495-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 22:59:59,312-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:00:00,077-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:00:09,320-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:00:19,329-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:00:28,818-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:00:29,337-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 23:00:35,687-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 23:00:39,345-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:00:49,353-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:00:59,362-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:01:01,141-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:01:09,370-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:01:11,297-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:01:19,378-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:01:29,386-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:01:33,464-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:01:39,394-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:01:46,907-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:01:49,403-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:01:59,411-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:02:05,787-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:02:09,419-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:02:19,427-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 23:02:22,517-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 23:02:29,435-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:02:38,110-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:02:39,444-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:02:49,452-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:02:58,127-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:02:59,460-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:03:09,468-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:03:10,433-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:03:19,476-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:03:29,485-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 23:03:33,737-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 23:03:39,493-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:03:42,756-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:03:49,500-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:03:59,508-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:04:09,347-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:04:09,516-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:04:15,079-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:04:19,524-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:04:29,532-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:04:39,540-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:04:44,957-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:04:47,402-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:04:49,548-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:04:59,556-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:05:09,564-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:05:19,572-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:05:19,725-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:05:20,567-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:05:29,580-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:05:39,587-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:05:49,595-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:05:52,048-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:05:56,177-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:05:59,603-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:06:09,611-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:06:19,619-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:06:24,371-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:06:29,627-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 23:06:31,787-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 23:06:39,635-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:06:49,643-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:06:56,694-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:06:59,651-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:07:07,397-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:07:09,658-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:07:19,666-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 23:07:29,017-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 23:07:29,674-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:07:39,682-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:07:43,007-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:07:49,690-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:07:59,698-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:08:01,340-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:08:09,706-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 23:08:18,617-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 23:08:19,714-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:08:29,721-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:08:33,663-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:08:39,729-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:08:49,737-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:08:54,227-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:08:59,746-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:09:05,986-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:09:09,754-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:09:19,762-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:09:29,770-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 23:09:29,837-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:09:38,309-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:09:39,778-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:09:49,786-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:09:59,794-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:10:05,447-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:10:09,802-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:10:10,632-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:10:19,810-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:10:29,818-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:10:39,827-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-05 23:10:41,058-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:10:42,955-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:10:49,835-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:10:59,843-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:11:09,851-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:11:15,278-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:11:16,667-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:11:19,859-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:11:29,867-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:11:39,875-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 23:11:47,601-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 23:11:49,883-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 23:11:52,277-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 23:11:59,892-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:12:09,900-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:12:19,908-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 23:12:19,924-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 23:12:27,887-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 23:12:29,916-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:12:39,924-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:12:49,932-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 23:12:52,247-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 23:12:59,940-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:13:03,497-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:13:09,949-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:13:19,957-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 23:13:24,570-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 23:13:29,965-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 23:13:39,107-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 23:13:39,973-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:13:49,981-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 23:13:56,892-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 23:13:59,989-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:14:09,997-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 23:14:14,718-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 23:14:20,005-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 23:14:29,215-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 23:14:30,013-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:14:40,021-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:14:50,029-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:14:50,328-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:15:00,038-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 23:15:01,538-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 23:15:10,046-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:15:20,054-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 23:15:25,939-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 23:15:30,062-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:15:33,862-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:15:40,071-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:15:50,079-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:16:00,087-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:16:01,549-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 23:16:06,185-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 23:16:10,095-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:16:20,104-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:16:30,112-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:16:37,159-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 23:16:38,507-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 23:16:40,120-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:16:50,129-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:17:00,137-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:17:10,145-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 23:17:10,831-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 23:17:12,769-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 23:17:20,153-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:17:30,162-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:17:40,170-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 23:17:43,156-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:17:48,379-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:17:50,178-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:18:00,187-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:18:10,195-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 23:18:15,479-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 23:18:20,203-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 23:18:23,989-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 23:18:30,211-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:18:40,219-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 23:18:47,803-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 23:18:50,228-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:18:59,599-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:19:00,236-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:19:10,244-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 23:19:20,126-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 23:19:20,252-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:19:30,261-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:19:35,209-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:19:40,269-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:19:50,277-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 23:19:52,450-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 23:20:00,285-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:20:10,293-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 23:20:10,819-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 23:20:20,302-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 23:20:24,775-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 23:20:30,310-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:20:40,318-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:20:46,429-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:20:50,326-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 23:20:57,097-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 23:21:00,334-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:21:10,342-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:21:20,350-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 23:21:22,040-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 23:21:29,420-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 23:21:30,357-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:21:40,365-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:21:50,373-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 23:21:57,649-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 23:22:00,381-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:22:01,744-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:22:10,389-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:22:20,396-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:22:30,404-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:22:33,259-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:22:34,067-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:22:40,412-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:22:50,420-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:23:00,428-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:23:06,390-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 23:23:08,869-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 23:23:10,435-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:23:20,443-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:23:30,451-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:23:38,713-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:23:40,459-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:23:44,479-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:23:50,467-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:24:00,474-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:24:10,482-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:24:11,036-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:24:20,089-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:24:20,490-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:24:30,498-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:24:40,506-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:24:43,359-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:24:50,514-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 23:24:55,699-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 23:25:00,521-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:25:10,529-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:25:15,682-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:25:20,537-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:25:30,545-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:25:31,309-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:25:40,552-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:25:48,005-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:25:50,560-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:26:00,568-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 23:26:06,919-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 23:26:10,576-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:26:20,328-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:26:20,584-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:26:30,592-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:26:40,599-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 23:26:42,530-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 23:26:50,607-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:26:52,651-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:27:00,615-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:27:10,623-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:27:18,140-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:27:20,631-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:27:24,974-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:27:30,639-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:27:40,646-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:27:50,654-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 23:27:53,750-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:27:57,297-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:28:00,662-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:28:10,670-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:28:20,678-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 23:28:29,361-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:28:29,620-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:28:30,686-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:28:40,694-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:28:50,702-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:29:00,710-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:29:01,943-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:29:04,971-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:29:10,717-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:29:20,725-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:29:30,733-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:29:34,266-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 23:29:40,580-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 23:29:40,741-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:29:50,749-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:30:00,757-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:30:06,589-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:30:10,766-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:30:16,191-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:30:20,774-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:30:30,782-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:30:38,912-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:30:40,790-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:30:50,798-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:30:51,801-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:31:00,806-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:31:10,814-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:31:11,235-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:31:20,823-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 23:31:27,411-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 23:31:30,831-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:31:40,839-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:31:43,558-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:31:50,847-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:32:00,855-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:32:03,021-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:32:10,862-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 23:32:15,881-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 23:32:20,870-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:32:30,878-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:32:38,631-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:32:40,886-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 23:32:48,203-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 23:32:50,893-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:33:00,901-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:33:10,909-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 23:33:14,241-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 23:33:20,526-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 23:33:20,917-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:33:30,924-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:33:40,932-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:33:49,851-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:33:50,940-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 23:33:52,850-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 23:34:00,948-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:34:10,955-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:34:20,963-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:34:25,174-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 23:34:25,461-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 23:34:30,971-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:34:40,979-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:34:50,986-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:34:57,497-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:35:00,994-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 23:35:01,072-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 23:35:11,002-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:35:21,009-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:35:29,820-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:35:31,017-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:35:36,682-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:35:41,025-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:35:51,033-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:36:01,040-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:36:02,143-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:36:11,048-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 23:36:12,293-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 23:36:21,056-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:36:31,064-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:36:34,466-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:36:41,072-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 23:36:47,903-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 23:36:51,079-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:37:01,087-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:37:06,789-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:37:11,095-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:37:21,103-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:37:23,513-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:37:31,110-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:37:39,112-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:37:41,118-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:37:51,126-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:37:59,123-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:38:01,134-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:38:11,142-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:38:11,435-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:38:21,149-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:38:31,157-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 23:38:34,733-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 23:38:41,165-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:38:43,758-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:38:51,173-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:39:01,180-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:39:10,343-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:39:11,188-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:39:16,081-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:39:21,196-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:39:31,204-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:39:41,212-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:39:45,953-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:39:48,404-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:39:51,219-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:40:01,227-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:40:11,235-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 23:40:20,727-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 23:40:21,243-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 23:40:21,563-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 23:40:31,251-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:40:41,259-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:40:51,266-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:40:53,051-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:40:57,173-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:41:01,274-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:41:11,282-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:41:21,290-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 23:41:25,373-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 23:41:31,298-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:41:32,784-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:41:41,306-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:41:51,313-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:41:57,697-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:42:01,321-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 23:42:08,393-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 23:42:11,329-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:42:21,337-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:42:30,020-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:42:31,345-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:42:41,352-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:42:44,003-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:42:51,360-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:43:01,368-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:43:02,343-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:43:11,376-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:43:19,613-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:43:21,384-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:43:31,392-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:43:34,666-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:43:41,399-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:43:51,407-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 23:43:55,223-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 23:44:01,415-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:44:06,989-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:44:11,423-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:44:21,431-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:44:30,833-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:44:31,439-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:44:39,312-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:44:41,447-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:44:51,455-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:45:01,463-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:45:06,443-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:45:11,471-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:45:11,635-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:45:21,479-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:45:31,486-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:45:41,494-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 23:45:42,053-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:45:43,958-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:45:51,502-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:46:01,510-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:46:11,518-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:46:16,281-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:46:17,663-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:46:21,525-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:46:31,533-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:46:41,541-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:46:48,604-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:46:51,549-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 23:46:53,273-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 23:47:01,556-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:47:11,564-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:47:20,927-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:47:21,572-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 23:47:28,883-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 23:47:31,580-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:47:41,588-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:47:51,595-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:47:53,250-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:48:01,603-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:48:04,493-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:48:11,611-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:48:21,619-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:48:25,573-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:48:31,627-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:48:40,103-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:48:41,635-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:48:51,642-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 23:48:57,895-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 23:49:01,650-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:49:11,658-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 23:49:15,713-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 23:49:21,666-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 23:49:30,218-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 23:49:31,674-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:49:41,681-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:49:51,323-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:49:51,689-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:50:01,697-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 23:50:02,542-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 23:50:11,705-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:50:21,713-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-05 23:50:26,934-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-05 23:50:31,720-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 23:50:34,866-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 23:50:41,728-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:50:51,736-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:51:01,744-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 23:51:02,543-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:51:07,190-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:51:11,752-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:51:21,759-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:51:31,767-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:51:38,153-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:51:39,513-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:51:41,775-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:51:51,783-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:52:01,791-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:52:11,798-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:52:11,836-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 23:52:13,763-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 23:52:21,806-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:52:31,814-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:52:41,822-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:52:44,159-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:52:49,373-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:52:51,830-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:53:01,837-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:53:11,845-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:53:16,482-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:53:21,853-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:53:24,983-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:53:31,861-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:53:41,868-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:53:48,805-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:53:51,876-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 23:54:00,593-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 23:54:01,884-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:54:11,892-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:54:21,128-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:54:21,900-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:54:31,907-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:54:36,203-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:54:41,915-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:54:51,923-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:54:53,451-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:55:01,931-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:55:11,813-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:55:11,939-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:55:21,946-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:55:25,774-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:55:31,954-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:55:41,962-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 23:55:47,423-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-05 23:55:51,970-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:55:58,097-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:56:01,977-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:56:11,985-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:56:21,993-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:56:23,033-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:56:30,420-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:56:32,001-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:56:42,009-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:56:52,016-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:56:58,643-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:57:02,024-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:57:02,743-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:57:12,032-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:57:22,040-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:57:32,047-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-05 23:57:34,253-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:57:35,066-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:57:42,055-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:57:52,063-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:58:02,071-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:58:07,389-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:58:09,863-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:58:12,079-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:58:22,087-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:58:32,095-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-05 23:58:39,711-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-05 23:58:42,103-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-05 23:58:45,473-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-05 23:58:52,111-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:59:02,119-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:59:12,035-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:59:12,127-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:59:21,084-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:59:22,135-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:59:32,143-0600 INFO Multiplexer No streams
-No streams
-2013-11-05 23:59:42,151-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:59:44,358-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-05 23:59:52,159-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-05 23:59:56,694-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:00:02,166-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:00:12,174-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:00:16,681-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:00:22,182-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:00:32,190-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:00:32,304-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:00:42,198-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:00:49,004-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:00:52,206-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:01:02,214-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:01:07,914-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:01:12,222-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:01:21,327-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:01:22,230-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:01:32,238-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:01:42,248-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:01:43,524-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:01:52,260-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:01:53,650-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:02:02,273-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:02:12,283-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:02:19,134-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:02:22,293-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:02:25,973-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:02:32,302-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:02:42,313-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:02:52,322-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 00:02:54,745-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 00:02:58,297-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 00:03:02,332-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:03:12,345-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:03:22,355-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 00:03:30,355-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:03:30,619-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:03:32,366-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:03:42,378-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:03:52,390-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:04:02,403-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:04:02,942-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 00:04:05,965-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 00:04:12,415-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:04:22,425-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:04:32,438-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 00:04:35,266-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:04:41,574-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:04:42,448-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:04:52,458-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:05:02,468-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:05:07,589-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:05:12,478-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 00:05:17,185-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 00:05:22,487-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:05:32,497-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:05:39,912-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:05:42,507-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:05:52,517-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:05:52,794-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:06:02,527-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:06:12,235-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:06:12,537-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:06:22,546-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 00:06:28,405-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 00:06:32,556-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:06:42,566-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 00:06:44,559-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 00:06:52,576-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:07:02,587-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 00:07:04,015-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 00:07:12,597-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 00:07:16,882-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 00:07:22,605-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:07:32,613-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:07:39,624-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:07:42,622-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 00:07:49,204-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 00:07:52,630-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:08:02,639-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:08:12,647-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:08:15,234-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 00:08:21,527-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 00:08:22,655-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:08:32,663-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:08:42,672-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:08:50,844-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:08:52,680-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-06 00:08:53,850-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-06 00:09:02,688-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:09:12,696-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:09:22,704-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:09:26,174-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:09:26,454-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:09:32,712-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:09:42,720-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:09:52,728-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:09:58,497-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:10:02,064-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:10:02,737-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:10:12,745-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:10:22,753-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:10:30,820-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:10:32,761-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:10:37,674-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:10:42,770-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:10:52,778-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:11:02,786-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:11:03,143-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:11:12,794-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:11:13,284-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:11:22,802-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:11:32,810-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:11:35,466-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:11:42,818-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:11:48,894-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:11:52,826-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:12:02,834-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:12:07,789-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:12:12,842-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:12:22,850-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:12:24,504-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:12:32,858-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:12:40,112-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:12:42,866-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:12:52,874-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:13:00,114-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:13:02,882-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:13:12,435-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:13:12,890-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:13:22,898-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:13:32,905-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:13:35,724-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:13:42,913-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:13:44,758-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:13:52,921-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:14:02,929-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:14:11,334-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:14:12,937-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-06 00:14:17,080-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-06 00:14:22,945-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:14:32,953-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:14:42,961-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:14:46,944-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-06 00:14:49,403-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-06 00:14:52,969-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:15:02,976-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:15:12,984-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:15:21,727-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:15:22,554-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:15:22,992-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:15:33,000-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:15:43,008-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:15:53,016-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:15:54,050-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 00:15:58,165-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 00:16:03,023-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:16:13,031-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:16:23,039-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:16:26,373-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:16:33,047-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:16:33,775-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:16:43,055-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:16:53,062-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:16:58,696-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:17:03,070-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:17:09,385-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:17:13,078-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:17:23,086-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:17:31,019-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:17:33,093-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:17:43,101-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:17:44,995-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:17:53,109-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:18:03,117-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:18:03,342-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:18:13,124-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:18:20,605-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:18:23,132-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:18:33,140-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:18:35,665-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:18:43,147-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:18:53,155-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:18:56,216-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:19:03,162-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:19:07,988-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:19:13,170-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:19:23,178-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:19:31,826-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:19:33,186-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:19:40,311-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:19:43,193-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:19:53,201-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:20:03,209-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:20:07,436-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:20:12,634-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:20:13,217-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:20:23,225-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:20:33,233-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:20:43,046-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:20:43,241-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:20:44,957-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:20:53,248-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:21:03,256-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:21:13,264-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:21:17,280-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:21:18,656-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:21:23,272-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:21:33,280-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:21:43,288-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:21:49,603-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:21:53,295-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-06 00:21:54,265-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-06 00:22:03,303-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:22:13,311-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:22:21,926-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:22:23,319-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:22:29,876-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:22:33,327-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:22:43,334-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:22:53,342-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:22:54,249-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:23:03,350-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:23:05,486-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:23:13,358-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:23:23,365-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:23:26,572-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:23:33,373-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:23:41,096-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:23:43,381-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:23:53,389-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:23:58,895-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:24:03,397-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:24:13,404-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:24:16,706-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:24:23,412-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:24:31,218-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:24:33,420-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:24:43,428-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:24:52,316-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:24:53,436-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:25:03,444-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:25:03,541-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:25:13,451-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:25:23,459-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:25:27,926-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:25:33,467-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:25:35,864-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:25:43,474-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:25:53,482-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:26:03,490-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 00:26:03,535-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:26:08,187-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:26:13,498-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:26:23,506-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:26:33,514-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:26:39,146-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:26:40,510-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:26:43,521-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:26:53,529-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:27:03,537-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:27:12,833-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:27:13,544-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 00:27:14,756-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 00:27:23,552-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:27:33,560-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:27:43,567-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:27:45,156-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:27:50,367-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:27:53,575-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:28:03,583-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:28:13,590-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:28:17,479-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:28:23,598-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:28:25,977-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:28:33,606-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:28:43,613-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:28:49,802-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:28:53,621-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:29:01,587-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:29:03,629-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:29:13,636-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:29:22,125-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:29:23,644-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:29:33,652-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:29:37,197-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:29:43,659-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:29:53,667-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:29:54,448-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:30:03,674-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:30:12,807-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:30:13,682-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:30:23,690-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 00:30:26,772-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 00:30:33,697-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:30:43,705-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:30:48,417-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:30:53,713-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 00:30:59,094-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 00:31:03,720-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:31:13,728-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:31:23,735-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-06 00:31:24,027-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-06 00:31:31,417-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-06 00:31:33,743-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:31:43,751-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:31:53,758-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:31:59,638-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 00:32:03,741-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 00:32:03,766-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:32:13,773-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:32:23,781-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:32:33,789-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:32:35,248-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-06 00:32:36,065-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-06 00:32:43,796-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:32:53,804-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:33:03,811-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:33:08,389-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:33:10,858-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:33:13,819-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:33:23,827-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:33:33,834-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:33:40,712-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:33:43,842-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:33:46,468-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:33:53,849-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:34:03,857-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:34:13,035-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:34:13,865-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-06 00:34:22,077-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-06 00:34:23,872-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:34:33,880-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:34:43,887-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 00:34:45,358-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 00:34:53,895-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:34:57,688-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:35:03,903-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:35:13,910-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 00:35:17,682-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 00:35:23,918-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:35:33,299-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:35:33,926-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:35:43,933-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:35:50,005-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:35:53,941-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:36:03,948-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:36:08,909-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:36:13,956-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:36:22,328-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:36:23,964-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:36:33,971-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:36:43,979-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:36:44,519-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:36:53,987-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:36:54,651-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:37:03,994-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:37:14,002-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:37:20,129-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:37:24,010-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:37:26,974-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:37:34,018-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:37:44,025-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:37:54,033-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:37:55,739-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:37:59,297-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:38:04,041-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:38:14,049-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:38:24,057-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:38:31,349-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-06 00:38:31,619-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-06 00:38:34,064-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:38:44,072-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:38:54,080-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 00:39:03,944-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 00:39:04,088-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:39:06,959-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:39:14,095-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:39:24,103-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:39:34,111-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-06 00:39:36,267-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:39:42,570-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:39:44,119-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:39:54,127-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:40:04,135-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 00:40:08,589-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 00:40:14,143-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:40:18,180-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:40:24,151-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:40:34,159-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:40:40,913-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:40:44,167-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:40:53,790-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:40:54,175-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:41:04,183-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:41:13,236-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:41:14,191-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:41:24,199-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:41:29,400-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:41:34,207-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:41:44,215-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:41:45,559-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:41:54,223-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:42:04,231-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:42:05,010-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:42:14,239-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:42:17,882-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:42:24,248-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:42:34,256-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:42:40,620-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:42:44,264-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-06 00:42:50,204-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-06 00:42:54,273-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:43:04,281-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:43:14,289-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:43:16,230-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-06 00:43:22,528-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-06 00:43:24,297-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:43:34,306-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:43:44,314-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:43:51,840-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:43:54,322-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-06 00:43:54,850-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-06 00:44:04,331-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:44:14,339-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:44:24,347-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 00:44:27,174-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-06 00:44:27,449-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-06 00:44:34,355-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:44:44,363-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:44:54,371-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 00:44:59,498-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:45:03,060-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:45:04,379-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:45:14,388-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:45:24,396-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:45:31,823-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:45:34,404-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 00:45:38,670-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 00:45:44,412-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:45:54,419-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 00:46:04,145-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 00:46:04,427-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:46:14,281-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:46:14,435-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:46:24,443-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:46:34,451-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 00:46:36,468-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 00:46:44,459-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:46:49,891-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:46:54,467-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:47:04,474-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:47:08,792-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:47:14,482-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:47:24,490-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 00:47:25,501-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 00:47:34,498-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:47:41,115-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:47:44,506-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:47:54,514-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-06 00:48:01,112-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-06 00:48:04,521-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:48:13,438-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:48:14,529-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:48:24,537-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:48:34,545-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:48:36,722-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:48:44,553-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:48:45,761-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:48:54,561-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:49:04,568-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:49:12,332-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:49:14,576-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:49:18,084-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:49:24,584-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:49:34,592-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:49:44,599-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:49:47,942-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:49:50,407-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:49:54,607-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:50:04,615-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:50:14,622-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:50:22,730-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:50:23,552-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:50:24,630-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:50:34,638-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:50:44,646-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:50:54,653-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:50:55,053-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:50:59,162-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:51:04,661-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:51:14,669-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:51:24,676-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:51:27,376-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:51:34,684-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:51:34,772-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:51:44,692-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:51:54,700-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:51:59,699-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:52:04,707-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:52:10,382-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:52:14,715-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:52:24,723-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:52:32,022-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:52:34,731-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:52:44,738-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:52:45,992-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:52:54,747-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 00:53:04,346-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 00:53:04,754-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:53:14,762-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 00:53:21,603-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 00:53:24,769-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:53:34,777-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 00:53:36,669-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 00:53:44,785-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:53:54,793-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:53:57,212-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:54:04,801-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:54:08,991-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:54:14,809-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:54:24,816-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:54:32,822-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:54:34,824-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:54:41,314-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:54:44,832-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:54:54,840-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:55:04,847-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:55:08,432-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:55:13,637-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:55:14,855-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:55:24,863-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:55:34,871-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:55:44,042-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:55:44,879-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:55:45,960-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:55:54,887-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:56:04,895-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:56:14,903-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:56:18,283-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:56:19,652-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:56:24,911-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:56:34,918-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:56:44,926-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:56:50,606-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:56:54,934-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 00:56:55,263-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 00:57:04,942-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:57:14,949-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:57:22,929-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:57:24,958-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 00:57:30,873-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 00:57:34,966-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:57:44,975-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:57:54,982-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:57:55,252-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:58:04,989-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:58:06,483-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:58:14,997-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:58:25,005-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:58:27,575-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:58:35,013-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:58:42,092-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:58:45,023-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:58:55,033-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:58:59,898-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:59:05,043-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:59:15,054-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:59:17,702-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:59:25,062-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 00:59:32,222-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 00:59:35,073-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 00:59:45,084-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 00:59:53,312-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 00:59:55,094-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:00:04,544-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:00:05,105-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:00:15,115-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:00:25,126-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:00:28,922-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:00:35,136-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 01:00:36,868-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 01:00:45,156-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:00:55,174-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 01:01:04,533-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 01:01:05,180-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:01:09,191-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:01:15,183-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:01:25,188-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:01:35,192-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 01:01:40,143-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 01:01:41,515-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 01:01:45,200-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:01:55,218-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:02:05,227-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:02:13,837-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:02:15,236-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:02:15,752-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:02:25,245-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:02:35,254-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:02:45,263-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 01:02:46,161-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 01:02:51,363-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 01:02:55,272-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:03:05,281-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:03:15,292-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:03:18,483-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:03:25,301-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 01:03:26,973-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 01:03:35,311-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:03:45,320-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:03:50,807-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:03:55,329-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:04:02,583-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:04:05,342-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:04:15,352-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:04:23,130-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:04:25,363-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:04:35,373-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:04:38,192-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:04:45,382-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:04:55,390-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:04:55,453-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:05:05,398-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:05:13,802-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:05:15,405-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:05:25,413-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:05:27,776-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:05:35,421-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:05:45,430-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 01:05:49,413-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 01:05:55,441-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 01:06:00,100-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 01:06:05,452-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:06:15,463-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 01:06:25,023-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 01:06:25,472-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 01:06:32,423-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 01:06:35,482-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:06:45,492-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:06:55,500-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:07:00,633-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 01:07:04,746-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 01:07:05,509-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:07:15,517-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:07:25,526-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:07:35,534-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:07:36,243-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:07:37,068-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:07:45,542-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:07:55,551-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:08:05,560-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 01:08:09,392-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:08:11,853-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:08:15,568-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:08:25,578-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:08:35,587-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 01:08:41,715-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 01:08:45,596-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 01:08:47,464-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 01:08:55,604-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:09:05,613-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 01:09:14,038-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 01:09:15,621-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:09:23,073-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:09:25,630-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:09:35,640-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:09:45,650-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:09:46,360-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:09:55,661-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:09:58,683-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:10:05,669-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:10:15,679-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:10:18,683-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:10:25,690-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:10:34,294-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:10:35,699-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:10:45,709-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:10:51,006-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:10:55,720-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:11:05,732-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:11:09,903-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:11:15,742-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 01:11:23,330-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 01:11:25,753-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:11:35,764-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:11:45,513-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:11:45,775-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:11:55,652-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:11:55,784-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:12:05,794-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:12:15,804-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 01:12:21,124-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 01:12:25,814-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:12:27,975-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:12:35,824-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:12:45,833-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:12:55,846-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:12:56,733-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 01:13:00,299-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 01:13:05,856-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:13:15,865-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:13:25,874-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:13:32,343-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:13:32,621-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:13:35,883-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:13:45,891-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:13:55,902-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 01:14:04,945-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 01:14:05,910-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:14:07,953-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:14:15,921-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:14:25,930-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:14:35,943-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 01:14:37,268-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 01:14:43,564-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 01:14:45,956-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:14:55,965-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:15:05,974-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:15:09,591-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:15:15,984-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:15:19,173-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:15:25,993-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:15:36,002-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 01:15:41,915-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 01:15:46,010-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:15:54,783-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:15:56,019-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:16:06,027-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:16:14,237-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:16:16,035-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:16:26,044-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:16:30,393-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:16:36,056-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:16:46,065-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:16:46,561-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:16:56,074-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 01:17:06,004-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 01:17:06,083-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:17:16,095-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 01:17:18,885-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 01:17:26,105-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:17:36,114-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:17:41,613-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:17:46,123-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:17:51,207-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:17:56,131-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:18:06,141-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:18:16,151-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:18:17,223-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:18:23,530-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:18:26,160-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:18:36,169-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:18:46,178-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:18:52,833-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:18:55,853-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:18:56,190-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:19:06,203-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:19:16,213-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:19:26,223-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:19:28,177-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:19:28,443-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:19:36,236-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:19:46,245-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:19:56,252-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:20:00,501-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:20:04,053-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:20:06,260-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:20:16,267-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:20:26,275-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:20:32,823-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:20:36,282-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:20:39,663-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:20:46,289-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:20:56,297-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:21:05,146-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:21:06,304-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:21:15,273-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:21:16,311-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:21:26,319-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:21:36,326-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:21:37,469-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:21:46,335-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:21:50,883-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:21:56,343-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:22:06,352-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:22:09,792-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:22:16,365-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:22:26,366-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:22:26,493-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:22:36,372-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 01:22:42,116-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 01:22:46,382-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:22:56,385-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:23:02,104-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:23:06,393-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:23:14,438-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:23:16,403-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:23:26,412-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:23:36,422-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 01:23:37,714-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 01:23:46,431-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:23:46,761-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:23:56,443-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:24:06,455-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 01:24:13,324-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 01:24:16,467-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:24:19,085-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:24:26,479-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:24:36,490-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:24:46,509-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:24:48,933-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:24:51,407-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:24:56,527-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:25:06,530-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:25:16,539-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 01:25:23,731-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:25:24,543-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:25:26,550-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:25:36,561-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:25:46,571-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:25:56,053-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:25:56,580-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:26:00,153-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:26:06,589-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:26:16,597-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:26:26,605-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:26:28,377-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:26:35,763-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:26:36,619-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:26:46,632-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:26:56,644-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 01:27:00,701-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 01:27:06,657-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 01:27:11,374-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 01:27:16,667-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:27:26,675-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:27:33,023-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:27:36,680-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:27:46,694-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 01:27:46,984-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 01:27:56,707-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:28:05,346-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:28:06,715-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:28:16,722-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 01:28:22,594-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 01:28:26,740-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:28:36,751-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:28:37,669-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:28:46,761-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:28:56,771-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:28:58,203-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:29:06,779-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:29:09,992-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:29:16,787-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:29:26,795-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 01:29:33,814-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 01:29:36,804-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:29:42,315-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:29:46,813-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:29:56,821-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:30:06,830-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 01:30:09,424-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 01:30:14,639-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 01:30:16,838-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:30:26,849-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:30:36,868-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 01:30:45,034-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 01:30:46,885-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:30:46,961-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:30:56,901-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:31:06,905-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:31:16,906-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:31:19,284-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 01:31:20,644-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 01:31:26,909-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:31:36,926-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:31:46,940-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 01:31:51,608-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 01:31:56,254-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 01:31:56,948-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:32:06,956-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:32:16,964-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 01:32:23,931-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 01:32:26,974-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 01:32:31,864-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 01:32:36,981-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:32:46,988-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 01:32:56,253-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 01:32:56,995-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:33:07,002-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:33:07,473-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:33:17,009-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:33:27,017-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 01:33:28,577-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 01:33:37,036-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:33:43,083-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:33:47,049-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:33:57,058-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 01:34:00,900-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 01:34:07,070-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:34:17,081-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 01:34:18,693-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 01:34:27,092-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 01:34:33,223-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 01:34:37,093-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:34:47,106-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:34:54,303-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:34:57,116-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:35:05,546-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:35:07,126-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:35:17,136-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:35:27,144-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 01:35:29,913-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 01:35:37,153-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:35:37,869-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:35:47,163-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:35:57,173-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 01:36:05,523-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 01:36:07,185-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:36:10,191-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:36:17,194-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:36:27,203-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:36:37,213-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:36:41,133-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 01:36:42,515-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 01:36:47,222-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:36:57,232-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:37:07,241-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 01:37:14,838-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 01:37:16,743-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 01:37:17,251-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:37:27,260-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:37:37,270-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:37:47,160-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:37:47,280-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 01:37:52,353-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 01:37:57,290-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:38:07,300-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:38:17,310-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 01:38:19,484-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 01:38:27,319-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 01:38:27,963-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 01:38:37,329-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:38:47,339-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 01:38:51,807-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 01:38:57,349-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 01:39:03,573-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 01:39:07,358-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:39:17,368-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 01:39:24,129-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 01:39:27,376-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:39:37,388-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 01:39:39,184-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 01:39:47,395-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:39:56,453-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:39:57,404-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:40:07,412-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:40:14,793-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:40:17,420-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:40:27,427-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:40:28,776-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:40:37,440-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:40:47,452-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 01:40:50,404-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 01:40:57,455-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 01:41:01,099-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 01:41:07,466-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:41:17,479-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 01:41:26,014-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 01:41:27,493-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:41:33,422-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:41:37,501-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:41:47,509-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:41:57,517-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 01:42:01,623-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:42:05,745-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:42:07,525-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:42:17,533-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:42:27,540-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 01:42:37,233-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 01:42:37,548-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-06 01:42:38,067-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-06 01:42:47,556-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:42:57,564-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:43:07,571-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 01:43:10,391-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:43:12,843-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:43:17,579-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:43:27,587-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:43:37,594-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 01:43:42,714-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 01:43:47,602-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 01:43:48,453-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 01:43:57,610-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:44:07,618-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 01:44:15,037-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 01:44:17,626-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 01:44:24,063-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 01:44:27,633-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:44:37,641-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 01:44:47,360-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 01:44:47,649-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:44:57,657-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:44:59,673-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:45:07,665-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:45:17,673-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 01:45:19,683-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 01:45:27,680-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 01:45:35,283-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 01:45:37,688-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:45:47,696-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:45:52,006-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:45:57,704-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:46:07,712-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 01:46:10,893-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 01:46:17,720-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:46:24,329-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:46:27,728-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:46:37,735-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:46:46,503-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:46:47,743-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:46:56,652-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:46:57,751-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:47:07,759-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:47:17,767-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:47:22,113-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:47:27,775-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:47:28,975-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:47:37,783-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:47:47,791-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 01:47:57,723-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 01:47:57,798-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:48:01,298-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:48:07,806-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:48:17,814-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:48:27,822-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:48:33,333-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:48:33,621-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:48:37,830-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:48:47,838-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:48:57,846-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:49:05,944-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:49:07,854-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:49:08,943-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:49:17,862-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:49:27,870-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:49:37,878-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:49:38,267-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 01:49:44,553-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 01:49:47,885-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:49:57,893-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:50:07,901-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 01:50:10,590-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 01:50:17,909-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:50:20,163-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:50:27,917-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:50:37,925-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 01:50:42,914-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 01:50:47,933-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:50:55,773-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:50:57,941-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:51:07,949-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 01:51:15,236-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 01:51:17,957-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:51:27,965-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 01:51:31,383-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 01:51:37,973-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 01:51:47,559-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 01:51:47,981-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:51:57,990-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-06 01:52:06,992-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-06 01:52:07,997-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:52:18,006-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:52:19,882-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:52:28,014-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:52:38,022-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:52:42,603-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:52:48,030-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:52:52,205-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:52:58,038-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:53:08,048-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:53:18,056-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 01:53:18,213-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:53:24,528-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:53:28,064-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:53:38,072-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:53:48,079-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:53:53,824-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:53:56,851-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:53:58,087-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:54:08,095-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:54:18,103-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:54:28,111-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:54:29,174-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:54:29,434-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:54:38,119-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:54:48,128-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:54:58,136-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:55:01,497-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 01:55:05,044-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 01:55:08,144-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:55:18,153-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:55:28,161-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:55:33,820-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:55:38,169-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:55:40,654-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:55:48,177-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:55:58,185-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:56:06,143-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:56:08,194-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:56:16,264-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:56:18,202-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:56:28,209-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:56:38,217-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:56:38,466-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:56:48,225-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 01:56:51,874-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 01:56:58,233-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:57:08,241-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:57:10,789-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:57:18,249-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:57:27,484-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:57:28,257-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:57:38,265-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:57:43,112-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:57:48,273-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:57:58,281-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:58:03,094-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:58:08,288-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:58:15,435-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:58:18,296-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:58:28,304-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:58:38,312-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 01:58:38,704-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-06 01:58:47,757-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-06 01:58:48,320-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:58:58,328-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:59:08,336-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:59:14,314-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 01:59:18,343-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-06 01:59:20,081-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-06 01:59:28,351-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:59:38,359-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 01:59:48,367-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 01:59:49,924-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-06 01:59:52,405-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-06 01:59:58,375-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:00:08,383-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:00:18,390-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 02:00:24,729-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-06 02:00:25,533-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-06 02:00:28,398-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:00:38,406-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:00:48,414-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:00:57,052-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:00:58,422-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:01:01,144-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:01:08,430-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:01:18,438-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:01:28,446-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:01:29,375-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:01:36,754-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:01:38,454-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:01:48,461-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:01:58,469-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:02:01,698-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:02:08,477-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-06 02:02:12,363-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-06 02:02:18,485-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:02:28,493-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:02:34,021-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:02:38,501-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:02:47,974-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:02:48,509-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:02:58,517-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:03:06,344-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:03:08,525-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:03:18,532-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:03:23,584-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:03:28,540-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:03:38,548-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:03:38,667-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:03:48,556-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:03:58,564-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-06 02:03:59,194-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-06 02:04:08,572-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:04:10,990-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:04:18,580-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:04:28,588-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:04:34,805-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:04:38,596-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:04:43,313-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:04:48,604-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:04:58,611-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:05:08,619-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:05:10,415-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:05:15,636-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:05:18,627-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:05:28,635-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:05:38,643-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:05:46,025-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:05:47,959-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:05:48,651-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:05:58,658-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:06:08,666-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:06:18,674-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:06:20,282-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:06:21,635-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:06:28,682-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:06:38,690-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:06:48,698-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:06:52,605-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-06 02:06:57,244-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-06 02:06:58,705-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:07:08,713-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:07:18,721-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:07:24,928-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:07:28,729-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:07:32,855-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:07:38,737-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:07:48,744-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:07:57,251-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:07:58,752-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 02:08:08,465-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 02:08:08,760-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:08:18,768-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:08:28,775-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:08:29,574-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:08:38,783-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:08:44,075-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:08:48,791-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:08:58,799-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:09:01,897-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:09:08,807-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:09:18,815-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:09:19,685-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:09:28,822-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:09:34,220-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:09:38,830-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:09:48,838-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-06 02:09:55,295-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-06 02:09:58,845-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:10:06,543-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:10:08,853-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:10:18,861-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:10:28,868-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:10:30,906-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:10:38,866-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:10:38,876-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:10:48,884-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:10:58,892-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:11:06,516-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:11:08,899-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:11:11,189-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:11:18,907-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:11:28,915-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:11:38,922-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-06 02:11:42,125-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:11:43,512-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:11:48,930-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:11:58,938-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:12:08,945-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:12:15,835-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:12:17,736-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:12:18,953-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:12:28,961-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:12:38,968-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:12:48,158-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:12:48,976-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:12:53,346-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:12:58,984-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:13:08,991-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:13:18,999-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:13:20,481-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:13:28,956-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:13:29,006-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:13:39,014-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:13:49,022-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:13:52,804-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:13:59,029-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:14:04,566-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:14:09,037-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:14:19,044-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:14:25,127-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:14:29,052-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:14:39,059-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:14:40,176-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:14:49,067-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:14:57,450-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:14:59,075-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:15:09,082-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:15:15,786-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:15:19,090-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:15:29,097-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:15:29,773-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:15:39,105-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:15:49,112-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:15:51,396-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:15:59,120-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:16:02,096-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:16:09,128-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:16:19,135-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:16:27,006-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:16:29,143-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:16:34,419-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:16:39,150-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:16:49,158-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:16:59,165-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 02:17:02,616-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:17:06,742-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:17:09,173-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:17:19,180-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:17:29,188-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:17:38,226-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:17:39,065-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:17:39,196-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:17:49,203-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:17:59,211-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:18:09,218-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:18:11,388-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:18:13,836-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:18:19,226-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:18:29,234-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:18:39,242-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:18:43,711-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:18:49,249-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:18:49,446-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:18:59,257-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:19:09,265-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:19:16,034-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:19:19,273-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:19:25,056-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:19:29,281-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:19:39,289-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:19:48,357-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:19:49,297-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:19:59,305-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:20:00,666-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:20:09,313-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:20:19,321-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:20:20,680-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:20:29,330-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:20:36,276-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:20:39,338-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:20:49,346-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:20:53,003-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:20:59,354-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:21:09,362-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:21:11,886-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:21:19,370-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:21:25,326-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:21:29,379-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:21:39,387-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:21:47,496-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:21:49,395-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:21:57,649-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:21:59,403-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:22:09,411-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:22:19,419-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:22:23,106-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:22:29,427-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:22:29,972-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:22:39,435-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:22:49,443-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 02:22:58,715-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 02:22:59,452-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:23:02,295-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:23:09,460-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:23:19,468-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:23:29,476-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:23:34,326-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:23:34,618-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:23:39,484-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:23:49,492-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:23:59,500-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:24:06,941-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:24:09,508-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 02:24:09,936-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 02:24:19,516-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:24:29,524-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:24:39,264-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:24:39,532-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:24:45,546-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:24:49,540-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:24:59,549-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:25:09,557-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:25:11,587-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:25:19,565-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:25:21,156-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:25:29,573-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:25:39,581-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:25:43,911-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:25:49,589-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:25:56,766-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:25:59,597-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:26:09,605-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:26:16,234-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:26:19,613-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:26:29,621-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:26:32,376-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:26:39,629-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:26:48,557-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:26:49,637-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:26:59,645-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:27:07,986-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:27:09,653-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:27:19,661-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:27:20,880-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:27:29,670-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:27:39,677-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 02:27:43,596-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 02:27:49,686-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:27:53,203-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:27:59,694-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:28:09,702-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:28:19,206-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:28:19,710-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:28:25,526-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:28:29,718-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:28:39,726-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:28:49,734-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:28:54,816-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:28:57,849-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:28:59,742-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:29:09,750-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:29:19,758-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:29:29,766-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:29:30,172-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 02:29:30,426-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 02:29:39,774-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:29:49,782-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:29:59,790-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:30:02,495-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:30:06,037-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:30:09,798-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:30:19,806-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:30:29,814-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:30:34,818-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:30:39,822-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 02:30:41,646-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 02:30:49,830-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:30:59,838-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:31:07,141-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:31:09,846-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:31:17,257-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:31:19,854-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:31:29,862-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:31:39,464-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:31:39,870-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:31:49,878-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 02:31:52,867-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 02:31:59,886-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:32:09,894-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:32:11,787-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:32:19,902-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:32:28,477-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:32:29,909-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:32:39,917-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:32:44,110-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:32:49,925-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:32:59,933-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:33:04,087-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:33:09,941-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:33:16,433-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:33:19,949-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:33:29,956-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 02:33:39,697-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 02:33:39,964-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:33:48,756-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:33:49,972-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:33:59,980-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:34:09,988-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:34:15,307-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:34:19,995-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:34:21,079-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:34:30,003-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:34:40,011-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:34:50,019-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 02:34:50,917-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:34:53,402-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:35:00,027-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:35:10,034-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:35:20,042-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:35:25,725-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:35:26,527-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:35:30,050-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:35:40,058-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:35:50,066-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:35:58,048-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:36:00,074-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 02:36:02,137-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 02:36:10,081-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:36:20,089-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:36:30,097-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:36:30,371-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:36:37,747-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:36:40,105-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:36:50,113-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:37:00,121-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 02:37:02,694-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 02:37:10,129-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 02:37:13,357-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 02:37:20,137-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:37:30,144-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 02:37:35,017-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 02:37:40,152-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:37:48,967-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:37:50,160-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:38:00,168-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 02:38:07,340-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 02:38:10,176-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:38:20,184-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 02:38:24,577-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 02:38:30,192-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 02:38:39,663-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 02:38:40,199-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:38:50,207-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:39:00,187-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:39:00,215-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:39:10,223-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:39:11,987-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:39:20,230-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:39:30,238-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 02:39:35,796-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 02:39:40,246-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:39:44,310-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:39:50,254-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:40:00,262-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:40:10,269-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 02:40:11,407-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:40:16,633-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:40:20,277-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:40:30,285-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:40:40,293-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:40:47,017-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 02:40:48,956-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 02:40:50,301-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:41:00,308-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:41:10,316-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:41:20,324-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 02:41:21,279-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 02:41:22,627-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 02:41:30,332-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:41:40,339-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:41:50,347-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 02:41:53,602-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:41:58,237-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:42:00,355-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:42:10,363-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:42:20,371-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 02:42:25,925-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 02:42:30,379-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 02:42:33,846-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 02:42:40,387-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:42:50,395-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 02:42:58,248-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 02:43:00,403-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:43:09,457-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:43:10,410-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:43:20,418-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:43:30,426-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 02:43:30,572-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 02:43:40,434-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:43:45,067-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:43:50,442-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:44:00,450-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:44:02,896-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:44:10,458-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:44:20,466-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 02:44:20,676-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 02:44:30,474-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 02:44:35,219-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 02:44:40,482-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:44:50,490-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 02:44:56,287-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 02:45:00,498-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:45:07,542-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:45:10,506-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:45:20,514-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:45:30,522-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 02:45:31,898-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 02:45:39,865-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 02:45:40,531-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:45:50,539-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:46:00,547-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 02:46:07,507-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 02:46:10,555-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 02:46:12,189-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 02:46:20,563-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:46:30,571-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:46:40,579-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:46:43,117-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 02:46:44,512-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 02:46:50,587-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:47:00,595-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:47:10,602-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:47:16,836-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 02:47:18,727-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 02:47:20,610-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:47:30,618-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:47:40,625-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:47:49,159-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:47:50,633-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:47:54,337-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:48:00,641-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:48:10,649-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:48:20,656-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:48:21,482-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:48:29,947-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:48:30,664-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:48:40,671-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:48:50,679-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:48:53,805-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:49:00,687-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 02:49:05,557-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 02:49:10,695-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:49:20,702-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:49:26,128-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:49:30,710-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:49:40,718-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:49:41,167-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:49:50,726-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:49:58,451-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:50:00,733-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:50:10,741-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:50:16,777-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:50:20,749-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:50:30,757-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:50:30,774-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:50:40,765-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:50:50,772-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 02:50:52,387-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 02:51:00,780-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:51:03,097-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:51:10,788-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:51:20,796-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:51:27,997-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:51:30,804-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:51:35,420-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:51:40,811-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:51:50,819-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:52:00,827-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 02:52:03,606-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:52:07,743-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:52:10,835-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:52:20,843-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:52:30,851-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:52:39,217-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:52:40,066-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:52:40,858-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:52:50,867-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:53:00,874-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:53:10,882-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:53:12,389-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 02:53:14,826-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 02:53:20,891-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:53:30,898-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:53:40,906-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:53:44,712-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 02:53:50,437-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 02:53:50,914-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:54:00,922-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:54:10,930-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:54:17,035-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:54:20,938-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 02:54:26,046-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 02:54:30,946-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:54:40,954-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 02:54:49,358-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 02:54:50,962-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:55:00,970-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:55:01,657-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:55:10,978-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:55:20,986-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 02:55:21,682-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 02:55:30,994-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 02:55:37,267-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 02:55:41,002-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:55:51,011-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:55:54,005-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:56:01,019-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:56:11,027-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 02:56:12,877-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 02:56:21,035-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:56:26,328-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:56:31,043-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:56:41,051-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 02:56:48,486-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 02:56:51,059-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:56:58,651-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:57:01,067-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:57:11,075-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:57:21,083-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:57:24,097-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:57:30,974-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:57:31,091-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:57:41,099-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:57:51,107-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 02:57:59,706-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 02:58:01,115-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:58:03,297-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:58:11,123-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:58:21,131-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:58:31,139-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 02:58:35,317-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 02:58:35,621-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 02:58:41,147-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:58:51,155-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:59:01,163-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:59:07,943-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 02:59:10,926-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 02:59:11,171-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:59:21,179-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 02:59:31,187-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 02:59:40,266-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 02:59:41,195-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 02:59:46,537-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 02:59:51,203-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:00:01,211-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:00:11,219-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:00:12,589-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:00:21,227-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 03:00:22,147-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 03:00:31,235-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:00:41,243-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 03:00:44,913-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 03:00:51,251-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:00:57,757-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:01:01,259-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:01:11,267-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 03:01:17,235-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 03:01:21,275-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:01:31,283-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 03:01:33,366-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 03:01:41,291-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:01:49,558-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:01:51,299-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:02:01,308-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 03:02:08,977-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 03:02:11,316-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:02:21,324-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:02:21,881-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:02:31,331-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:02:41,340-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 03:02:44,586-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 03:02:51,348-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:02:54,204-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:03:01,356-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:03:11,364-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:03:20,197-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:03:21,372-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:03:26,527-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:03:31,380-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:03:41,388-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:03:51,396-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 03:03:55,807-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:03:58,850-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:04:01,404-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:04:11,412-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:04:21,420-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:04:31,173-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:04:31,417-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:04:31,428-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:04:41,436-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:04:51,444-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:05:01,452-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:05:03,496-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 03:05:07,027-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 03:05:11,460-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:05:21,468-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:05:31,476-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 03:05:35,820-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 03:05:41,484-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 03:05:42,636-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 03:05:51,492-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:06:01,500-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:06:08,143-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:06:11,507-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 03:06:18,247-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 03:06:21,516-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:06:31,524-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:06:40,466-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:06:41,532-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:06:51,540-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 03:06:53,856-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 03:07:01,548-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:07:11,556-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:07:12,789-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:07:21,564-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 03:07:29,467-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 03:07:31,572-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:07:41,580-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:07:45,112-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:07:51,588-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:08:01,597-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 03:08:05,077-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 03:08:11,605-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:08:17,435-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:08:21,613-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:08:31,621-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 03:08:40,688-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 03:08:41,629-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:08:49,758-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:08:51,637-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:09:01,645-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:09:11,653-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 03:09:16,298-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 03:09:21,661-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:09:22,081-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:09:31,668-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:09:41,677-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:09:51,684-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 03:09:51,909-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:09:54,404-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:10:01,692-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:10:11,700-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:10:21,708-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:10:26,727-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 03:10:27,519-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 03:10:31,716-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:10:41,724-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:10:51,732-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 03:10:59,050-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 03:11:01,740-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:11:03,129-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:11:11,748-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:11:21,756-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 03:11:31,373-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 03:11:31,764-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 03:11:38,738-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 03:11:41,772-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:11:51,780-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:12:01,788-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 03:12:03,697-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 03:12:11,796-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:12:14,349-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:12:21,804-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:12:31,812-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 03:12:36,020-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 03:12:41,820-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 03:12:49,959-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 03:12:51,828-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:13:01,836-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 03:13:08,343-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 03:13:11,844-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:13:21,852-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 03:13:25,570-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 03:13:31,860-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 03:13:40,666-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 03:13:41,867-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:13:51,875-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:14:01,180-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:14:01,883-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:14:11,891-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:14:12,989-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:14:21,899-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:14:31,907-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 03:14:36,790-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 03:14:41,915-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:14:45,312-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:14:51,923-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:15:01,931-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:15:11,939-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:15:12,400-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:15:17,635-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:15:21,947-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:15:31,955-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:15:41,963-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 03:15:48,010-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:15:49,958-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:15:51,971-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:16:01,979-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:16:11,987-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:16:21,995-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:16:22,281-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 03:16:23,619-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 03:16:32,002-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:16:42,010-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:16:52,018-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:16:54,604-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:16:59,230-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:17:02,026-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:17:12,034-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:17:22,042-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:17:26,927-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:17:32,050-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 03:17:34,839-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 03:17:42,058-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:17:52,066-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 03:17:59,249-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 03:18:02,074-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:18:10,450-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:18:12,076-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:18:22,084-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 03:18:31,573-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 03:18:32,092-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:18:42,100-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:18:46,060-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:18:52,108-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:19:02,116-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 03:19:03,895-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 03:19:12,124-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 03:19:21,670-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 03:19:22,132-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:19:32,140-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 03:19:36,219-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 03:19:42,148-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:19:52,157-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:19:57,280-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:20:02,164-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 03:20:08,542-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 03:20:12,173-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:20:22,181-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:20:32,189-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:20:32,890-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:20:40,865-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:20:42,197-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:20:52,205-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:21:02,213-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:21:08,500-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:21:12,221-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:21:13,188-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:21:22,229-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:21:32,237-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:21:42,245-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 03:21:44,110-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:21:45,511-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:21:52,253-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:22:02,261-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:22:12,269-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:22:17,834-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:22:19,720-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:22:22,277-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:22:32,285-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:22:42,292-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:22:50,157-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:22:52,300-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 03:22:55,330-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 03:23:02,308-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:23:12,316-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:23:22,324-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:23:22,480-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:23:30,940-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:23:32,331-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:23:42,339-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:23:52,347-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:23:54,803-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:24:02,355-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 03:24:06,549-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 03:24:12,363-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:24:22,371-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:24:27,126-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:24:32,379-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:24:42,160-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:24:42,387-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:24:52,395-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:24:59,449-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:25:02,403-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:25:12,411-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 03:25:17,770-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 03:25:22,419-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 03:25:31,773-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 03:25:32,427-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:25:42,435-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:25:52,443-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:25:53,380-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:26:02,452-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 03:26:04,095-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 03:26:12,460-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:26:22,468-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 03:26:28,989-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 03:26:32,476-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 03:26:36,418-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 03:26:42,484-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:26:52,492-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:27:02,500-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:27:04,600-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:27:08,741-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:27:12,509-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:27:22,517-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:27:32,525-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 03:27:40,211-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:27:41,064-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:27:42,533-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:27:52,541-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:28:02,550-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:28:12,558-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 03:28:13,387-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:28:15,821-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:28:22,565-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:28:32,574-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:28:42,582-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:28:45,710-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:28:51,431-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:28:52,590-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:29:02,598-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:29:12,607-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:29:18,033-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:29:22,615-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 03:29:27,041-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 03:29:32,624-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:29:42,633-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:29:50,356-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:29:52,643-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:30:02,652-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:30:02,660-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:30:12,666-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:30:22,669-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:30:22,679-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:30:32,670-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:30:38,262-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:30:42,680-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:30:52,690-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:30:55,003-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:31:02,703-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:31:12,713-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:31:13,872-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:31:22,723-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 03:31:27,327-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 03:31:32,733-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:31:42,744-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:31:49,482-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:31:52,756-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:31:59,649-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:32:02,766-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:32:12,776-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:32:22,785-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 03:32:25,092-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:32:31,972-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:32:32,793-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:32:42,801-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:32:52,809-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:33:00,702-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:33:02,817-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:33:04,295-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:33:12,825-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:33:22,832-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:33:32,843-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:33:36,312-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:33:36,618-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:33:42,855-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:33:52,871-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:34:02,879-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:34:08,941-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:34:11,922-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:34:12,889-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:34:22,896-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:34:32,904-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:34:41,264-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:34:42,912-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:34:47,532-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:34:52,922-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:35:02,930-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:35:12,938-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 03:35:13,588-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 03:35:22,947-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:35:23,142-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:35:32,955-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:35:42,962-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:35:45,910-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:35:52,970-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:35:58,752-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:36:02,978-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:36:12,986-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:36:18,233-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:36:22,994-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:36:33,003-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 03:36:34,363-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 03:36:43,013-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:36:50,556-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:36:53,023-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:37:03,033-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:37:09,972-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:37:13,044-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:37:22,879-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:37:23,054-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:37:33,064-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:37:43,072-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 03:37:45,583-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 03:37:53,082-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 03:37:55,203-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 03:38:03,092-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:38:13,102-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:38:21,192-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:38:23,111-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:38:27,525-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:38:33,121-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:38:43,130-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:38:53,140-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:38:56,802-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:38:59,848-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:39:03,150-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:39:13,159-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:39:23,169-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 03:39:32,172-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 5ms
-2013-11-06 03:39:32,416-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 5ms
-No streams
-2013-11-06 03:39:33,179-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:39:43,188-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:39:53,198-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:40:03,206-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 03:40:04,495-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:40:08,023-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:40:13,216-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:40:23,227-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:40:33,241-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 03:40:36,818-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 03:40:43,251-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:40:43,634-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:40:53,259-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:41:03,270-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:41:09,141-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:41:13,278-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 03:41:19,245-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 03:41:23,286-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:41:33,295-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:41:41,464-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:41:43,303-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:41:53,311-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:41:54,854-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:42:03,318-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:42:13,326-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 03:42:13,788-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 03:42:23,335-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 03:42:30,464-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 03:42:33,343-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:42:43,350-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 03:42:46,111-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 03:42:53,359-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:43:03,366-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:43:06,075-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:43:13,374-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:43:18,434-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:43:23,383-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:43:33,390-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:43:41,685-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:43:43,398-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 03:43:50,757-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 03:43:53,405-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:44:03,412-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:44:13,420-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:44:17,295-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:44:23,079-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:44:23,427-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:44:33,434-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:44:43,442-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:44:52,905-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:44:53,450-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 03:44:55,403-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 03:45:03,458-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:45:13,466-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:45:23,473-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:45:27,725-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 03:45:28,516-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 03:45:33,481-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:45:43,489-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:45:53,497-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:46:00,048-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:46:03,505-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 03:46:04,125-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 03:46:13,513-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:46:23,520-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:46:32,371-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:46:33,528-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:46:39,736-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:46:43,536-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:46:53,544-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:47:03,552-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:47:04,694-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:47:13,559-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:47:15,346-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:47:23,567-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:47:33,575-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:47:37,017-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:47:43,583-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:47:50,956-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:47:53,591-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:48:03,598-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:48:09,340-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:48:13,606-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:48:23,614-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:48:26,566-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:48:33,622-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:48:41,663-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:48:43,630-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:48:53,637-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:49:02,176-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:49:03,645-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:49:13,653-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:49:13,986-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:49:23,661-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:49:33,669-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:49:37,787-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:49:43,677-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:49:46,309-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:49:53,685-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:50:03,693-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:50:13,397-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:50:13,701-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:50:18,632-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:50:23,709-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:50:33,717-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:50:43,725-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:50:49,007-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:50:50,955-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:50:53,734-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:51:03,742-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:51:13,750-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:51:23,278-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:51:23,758-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:51:24,617-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:51:33,766-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:51:43,774-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:51:53,782-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:51:55,602-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:52:00,227-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:52:03,790-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:52:13,798-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:52:23,807-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 03:52:27,926-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 03:52:33,815-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 03:52:35,837-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 03:52:43,823-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:52:53,831-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:53:00,249-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:53:03,839-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:53:11,448-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:53:13,847-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:53:23,855-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 03:53:32,573-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 03:53:33,863-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:53:43,871-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 03:53:47,059-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 03:53:53,879-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:54:03,888-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:54:04,895-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:54:13,896-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 03:54:22,668-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 03:54:23,904-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:54:33,912-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 3ms
-2013-11-06 03:54:37,220-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 3ms
-No streams
-2013-11-06 03:54:43,920-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:54:53,927-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:54:58,279-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:55:03,935-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 03:55:09,542-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 03:55:13,943-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:55:23,951-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 03:55:33,890-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 03:55:33,959-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 03:55:41,865-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 03:55:43,967-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:55:53,974-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:56:03,982-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:56:09,499-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:56:13,990-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 03:56:14,188-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 03:56:23,998-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:56:34,006-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:56:44,014-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:56:45,109-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 03:56:46,511-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 03:56:54,022-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:57:04,030-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:57:14,038-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:57:18,833-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:57:20,719-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:57:24,046-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:57:34,054-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:57:44,062-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 03:57:51,157-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 03:57:54,070-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:57:56,329-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:58:04,078-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:58:14,087-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:58:23,479-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:58:24,095-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:58:31,939-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:58:34,103-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:58:44,111-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:58:54,119-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:58:55,802-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:59:04,127-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:59:07,549-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:59:14,135-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:59:24,143-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:59:28,125-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:59:34,151-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 03:59:43,159-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 03:59:44,160-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 03:59:54,168-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:00:00,448-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:00:04,176-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:00:14,184-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:00:18,769-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:00:24,192-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:00:32,771-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:00:34,200-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:00:44,209-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:00:54,217-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:00:54,379-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:01:04,225-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:01:05,094-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:01:14,233-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:01:24,242-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:01:29,989-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:01:34,250-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:01:37,417-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:01:44,258-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:01:54,266-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:02:04,274-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:02:05,599-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 04:02:09,741-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 04:02:14,283-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:02:24,291-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:02:34,299-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:02:41,209-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:02:42,063-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:02:44,307-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:02:54,315-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:03:04,323-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:03:14,331-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:03:14,386-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:03:16,819-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:03:24,339-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:03:34,347-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:03:44,356-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 19ms
-2013-11-06 04:03:46,727-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 19ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 04:03:52,429-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 04:03:54,364-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:04:04,372-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:04:14,380-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:04:19,032-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:04:24,388-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:04:28,040-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:04:34,396-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:04:44,404-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:04:51,355-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:04:54,412-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 04:05:03,651-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 04:05:04,420-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:05:14,428-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:05:23,678-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:05:24,436-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:05:34,444-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:05:39,260-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:05:44,452-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:05:54,460-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:05:56,001-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:06:04,468-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:06:14,476-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:06:14,870-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:06:24,484-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:06:28,324-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:06:34,492-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:06:44,500-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:06:50,480-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:06:54,508-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 04:07:00,648-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 04:07:04,516-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:07:14,524-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:07:24,532-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:07:26,090-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:07:32,970-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:07:34,540-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:07:44,548-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:07:54,557-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 04:08:01,701-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 04:08:04,565-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:08:05,293-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:08:14,573-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:08:24,582-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:08:34,590-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 04:08:37,310-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:08:37,616-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:08:44,598-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:08:54,606-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:09:04,614-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 04:09:09,940-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:09:12,921-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:09:14,623-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:09:24,631-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:09:34,639-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 04:09:42,263-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 04:09:44,647-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:09:48,531-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:09:54,656-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:10:04,664-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 04:10:14,586-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 04:10:14,672-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:10:24,141-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:10:24,680-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:10:34,688-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:10:44,696-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 04:10:46,909-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 04:10:54,704-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:10:59,751-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:11:04,712-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:11:14,720-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 3ms
-2013-11-06 04:11:19,233-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 3ms
-No streams
-2013-11-06 04:11:24,728-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:11:34,736-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:11:35,361-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:11:44,745-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 04:11:51,555-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 04:11:54,753-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:12:04,761-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 174ms
-2013-11-06 04:12:11,144-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 174ms
-No streams
-2013-11-06 04:12:14,769-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 04:12:23,878-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 04:12:24,777-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:12:34,785-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:12:44,793-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:12:46,581-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:12:54,801-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 04:12:56,201-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 04:13:04,809-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:13:14,818-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 04:13:22,191-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 04:13:24,826-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 04:13:28,524-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 04:13:34,834-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:13:44,842-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:13:54,850-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:13:57,802-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 04:14:00,847-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 04:14:04,858-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:14:14,866-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:14:24,874-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:14:33,169-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 6ms
-2013-11-06 04:14:33,417-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 6ms
-No streams
-2013-11-06 04:14:34,882-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:14:44,890-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:14:54,898-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:15:04,906-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:15:05,492-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:15:09,022-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:15:14,914-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:15:24,922-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:15:34,930-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:15:37,815-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:15:44,632-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:15:44,938-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:15:54,946-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:16:04,954-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:16:10,138-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:16:14,962-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:16:20,243-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:16:24,970-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:16:34,978-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:16:42,461-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:16:44,986-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:16:54,994-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:16:55,853-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:17:05,002-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 04:17:14,785-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 04:17:15,010-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:17:25,018-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:17:31,463-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:17:35,027-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:17:45,035-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:17:47,107-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:17:55,043-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:18:05,052-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:18:07,073-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:18:15,060-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:18:19,430-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:18:25,068-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:18:35,076-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:18:42,683-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:18:45,084-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:18:51,753-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:18:55,092-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:19:05,100-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:19:15,109-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:19:18,293-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:19:24,076-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:19:25,117-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:19:35,126-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:19:45,134-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:19:53,903-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:19:55,142-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:19:56,399-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:20:05,150-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:20:15,158-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:20:25,166-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:20:28,722-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 04:20:29,514-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 04:20:35,175-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:20:45,183-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:20:55,192-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:21:01,046-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:21:05,123-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:21:05,200-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:21:15,208-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:21:25,216-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:21:33,369-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:21:35,224-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 04:21:40,733-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 04:21:45,233-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:21:55,241-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:22:05,250-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:22:05,693-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:22:15,258-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:22:16,344-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:22:25,266-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:22:35,275-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 04:22:38,017-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 04:22:45,283-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:22:51,954-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:22:55,291-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:23:05,299-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:23:10,339-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:23:15,307-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:23:25,314-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:23:27,564-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:23:35,322-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:23:42,663-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:23:45,330-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:23:55,338-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:24:03,174-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:24:05,346-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:24:14,986-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:24:15,354-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:24:25,362-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:24:35,370-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-2013-11-06 04:24:38,785-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-No streams
-2013-11-06 04:24:45,377-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:24:47,309-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:24:55,385-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:25:05,393-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:25:14,394-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:25:15,401-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:25:19,632-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:25:25,409-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:25:35,417-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:25:45,424-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 04:25:50,005-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:25:51,955-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:25:55,432-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:26:05,440-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:26:15,449-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:26:24,278-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:26:25,457-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:26:25,614-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:26:35,465-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:26:45,473-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:26:55,481-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:26:56,601-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:27:01,224-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:27:05,489-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:27:15,497-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:27:25,506-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:27:28,925-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:27:35,514-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 04:27:36,833-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 04:27:45,523-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:27:55,531-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:28:01,248-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:28:05,539-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:28:12,444-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:28:15,547-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:28:25,556-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:28:33,571-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:28:35,564-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:28:45,572-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 04:28:48,055-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 04:28:55,581-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:29:05,589-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:29:05,894-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:29:15,598-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:29:23,664-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:29:25,606-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:29:35,614-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:29:38,217-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:29:45,622-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:29:55,631-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:29:59,274-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:30:05,639-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:30:10,540-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:30:15,647-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:30:25,656-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:30:34,884-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:30:35,664-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:30:42,863-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:30:45,672-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:30:55,680-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:31:05,689-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:31:10,494-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 04:31:15,186-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 04:31:15,697-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:31:25,705-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:31:35,714-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:31:45,722-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:31:46,104-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:31:47,509-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:31:55,730-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:32:05,738-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:32:15,746-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:32:19,833-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:32:21,714-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:32:25,754-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:32:35,763-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:32:45,771-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:32:52,157-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:32:55,779-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 04:32:57,325-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 04:33:05,787-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:33:15,796-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:33:24,481-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:33:25,804-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:33:32,934-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:33:35,812-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:33:45,820-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:33:55,828-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:33:56,805-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:34:05,836-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:34:08,544-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:34:15,844-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:34:25,853-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:34:29,129-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:34:35,861-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:34:44,154-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:34:45,869-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:34:55,877-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 04:35:01,453-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 04:35:05,886-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:35:15,894-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:35:19,764-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:35:25,902-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 04:35:33,778-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 04:35:35,911-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:35:45,919-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:35:55,374-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:35:55,927-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:36:05,935-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 04:36:06,100-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 04:36:15,943-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:36:25,952-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 04:36:30,984-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 04:36:35,960-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:36:38,423-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:36:45,968-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:36:55,976-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:37:05,985-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:37:06,595-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 04:37:10,747-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 04:37:15,993-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:37:26,001-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:37:36,009-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 5ms
-2013-11-06 04:37:42,209-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 5ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:37:43,069-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:37:46,018-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:37:56,026-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:38:06,034-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:38:15,392-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:38:16,042-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:38:17,815-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:38:26,050-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:38:36,059-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:38:46,067-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 04:38:47,716-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:38:53,425-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:38:56,075-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:39:06,083-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:39:16,092-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:39:20,038-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:39:26,100-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:39:29,035-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:39:36,108-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:39:46,116-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:39:52,361-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:39:56,124-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:40:04,645-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:40:06,132-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:40:16,141-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 04:40:24,685-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 04:40:26,149-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:40:36,157-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:40:40,256-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:40:46,165-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:40:56,173-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:40:57,007-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:41:06,182-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:41:15,865-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:41:16,190-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:41:26,198-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 04:41:29,331-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 04:41:36,206-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:41:46,214-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:41:51,475-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:41:56,222-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:42:01,653-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:42:06,230-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:42:16,238-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:42:26,246-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:42:27,085-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 04:42:33,977-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 04:42:36,254-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:42:46,263-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:42:56,271-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 04:43:02,696-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 04:43:06,280-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 04:43:06,300-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 04:43:16,288-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:43:26,296-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:43:36,304-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:43:38,305-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:43:38,622-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:43:46,312-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:43:56,321-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:44:06,329-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:44:10,945-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 04:44:13,915-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 04:44:16,337-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:44:26,346-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:44:36,354-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:44:43,268-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:44:46,362-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:44:49,525-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:44:56,370-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:45:06,378-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:45:15,591-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:45:16,387-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 04:45:25,136-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 04:45:26,395-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:45:36,403-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:45:46,412-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 04:45:47,915-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 04:45:56,420-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:46:00,745-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:46:06,428-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:46:16,436-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 04:46:20,238-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 04:46:26,444-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:46:36,355-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:46:36,452-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:46:46,460-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:46:52,560-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:46:56,469-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:47:06,477-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:47:11,965-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:47:16,485-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:47:24,883-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:47:26,493-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:47:36,501-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:47:46,509-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:47:47,575-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:47:56,517-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:47:57,206-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:48:06,525-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:48:16,533-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:48:23,185-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:48:26,541-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:48:29,529-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:48:36,549-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:48:46,557-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:48:56,566-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:48:58,795-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:49:01,852-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:49:06,574-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:49:16,582-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:49:26,590-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:49:34,175-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:49:34,405-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:49:36,598-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:49:46,606-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:49:56,614-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:50:06,498-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:50:06,622-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:50:10,015-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:50:16,630-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:50:26,638-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:50:36,647-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 04:50:38,822-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:50:45,625-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:50:46,654-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:50:56,662-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:51:06,671-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:51:11,144-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:51:16,678-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:51:21,235-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:51:26,686-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:51:36,695-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:51:43,467-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:51:46,703-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:51:56,711-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:51:56,845-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:52:06,719-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:52:15,790-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:52:16,728-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:52:26,736-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:52:32,455-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:52:36,744-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:52:46,752-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:52:48,113-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:52:56,761-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:53:06,769-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 04:53:08,065-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 04:53:16,777-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:53:20,436-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:53:26,785-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:53:36,793-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:53:43,675-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:53:46,801-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:53:52,759-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:53:56,809-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:54:06,817-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:54:16,826-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:54:19,285-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:54:25,082-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:54:26,834-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:54:36,842-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:54:46,850-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:54:54,895-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:54:56,858-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:54:57,405-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:55:06,866-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:55:16,875-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:55:26,883-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 04:55:29,729-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 8ms
-2013-11-06 04:55:30,512-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 8ms
-No streams
-2013-11-06 04:55:36,891-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:55:46,899-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:55:56,907-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:56:02,051-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:56:06,115-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:56:06,915-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:56:16,923-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:56:26,932-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:56:34,374-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:56:36,940-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:56:41,725-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:56:46,948-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:56:56,955-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:57:06,697-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:57:06,964-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:57:16,972-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:57:17,336-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:57:26,980-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:57:36,988-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:57:39,020-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:57:46,996-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:57:52,946-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:57:57,004-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:58:07,012-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:58:11,343-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:58:17,020-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:58:27,028-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:58:28,556-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:58:37,036-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:58:43,666-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:58:47,044-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:58:57,052-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:59:04,166-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:59:07,061-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:59:15,989-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:59:17,069-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:59:27,077-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 04:59:37,084-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-2013-11-06 04:59:39,778-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-No streams
-2013-11-06 04:59:47,092-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 04:59:48,312-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 04:59:57,101-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:00:07,109-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 05:00:15,387-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 05:00:17,117-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:00:20,635-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:00:27,125-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:00:37,133-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:00:47,141-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:00:50,996-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 3ms
-2013-11-06 05:00:52,960-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 3ms
-No streams
-2013-11-06 05:00:57,151-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:01:07,159-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:01:17,168-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:01:25,281-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:01:26,606-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:01:27,176-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:01:37,184-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:01:47,192-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:01:57,201-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:01:57,604-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:02:02,216-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:02:07,209-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:02:17,217-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:02:27,226-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:02:29,927-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:02:37,234-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:02:37,826-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:02:47,242-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:02:57,250-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:03:02,250-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:03:07,258-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:03:13,436-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:03:17,266-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:03:27,274-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:03:34,573-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:03:37,282-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:03:47,290-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:03:49,046-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:03:57,299-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:04:06,896-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:04:07,307-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:04:17,315-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:04:24,656-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:04:27,323-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:04:37,331-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:04:39,219-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:04:47,339-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:04:57,347-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 05:05:00,266-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 05:05:07,355-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:05:11,542-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:05:17,363-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:05:27,371-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:05:35,876-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:05:37,380-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:05:43,865-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:05:47,388-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:05:57,396-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:06:07,404-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 05:06:11,487-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:06:16,188-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:06:17,412-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:06:27,420-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:06:37,428-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:06:47,096-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:06:47,436-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:06:48,511-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:06:57,444-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:07:07,453-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:07:17,461-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 05:07:20,835-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:07:22,706-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:07:27,469-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:07:37,477-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:07:47,485-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:07:53,157-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:07:57,493-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 05:07:58,317-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 05:08:07,501-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:08:17,510-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 05:08:25,480-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 05:08:27,518-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:08:33,926-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:08:37,526-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:08:47,534-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:08:57,543-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:08:57,803-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:09:07,551-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 05:09:09,537-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 05:09:17,559-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:09:27,567-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:09:30,126-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:09:37,575-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:09:45,146-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:09:47,582-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:09:57,589-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:10:02,449-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:10:07,598-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:10:17,606-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 05:10:20,757-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 05:10:27,614-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:10:34,772-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:10:37,623-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:10:47,631-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:10:56,366-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:10:57,639-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:11:07,095-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:11:07,647-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:11:17,655-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:11:27,663-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:11:31,976-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:11:37,672-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:11:39,418-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:11:47,680-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:11:57,688-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 05:12:07,586-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 05:12:07,696-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 05:12:11,742-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 05:12:17,704-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:12:27,713-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:12:37,721-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:12:43,197-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:12:44,064-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:12:47,729-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:12:57,737-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:13:07,745-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:13:16,387-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:13:17,753-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:13:18,807-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:13:27,761-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:13:37,769-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:13:47,777-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:13:48,710-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:13:54,417-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:13:57,785-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:14:07,793-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:14:17,802-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:14:21,033-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:14:27,810-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:14:30,027-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:14:37,818-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:14:47,826-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 05:14:53,356-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 05:14:57,834-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 05:15:05,638-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 05:15:07,842-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:15:17,850-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:15:25,679-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:15:27,858-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:15:37,866-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:15:41,247-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:15:47,874-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:15:57,883-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:15:58,002-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:16:07,891-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:16:16,857-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:16:17,899-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:16:27,907-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:16:30,325-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:16:37,915-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:16:47,923-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 05:16:52,467-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 05:16:57,931-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:17:02,648-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:17:07,939-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:17:17,947-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:17:27,955-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:17:28,077-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:17:34,971-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:17:37,963-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:17:47,971-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:17:57,979-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 05:18:03,688-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:18:07,294-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:18:07,987-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:18:17,995-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:18:28,003-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:18:38,011-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 05:18:39,297-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:18:39,617-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:18:48,018-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:18:58,027-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:19:08,035-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 05:19:11,940-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:19:14,907-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:19:18,043-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:19:28,051-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:19:38,059-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 05:19:44,263-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 05:19:48,067-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:19:50,517-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:19:58,075-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:20:08,084-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 05:20:16,586-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 05:20:18,092-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:20:26,127-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:20:28,100-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:20:38,108-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:20:48,116-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 05:20:48,909-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 05:20:58,124-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:21:01,737-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:21:08,133-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:21:18,141-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:21:21,232-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:21:28,149-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 05:21:37,347-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 05:21:38,157-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:21:48,165-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 05:21:53,555-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 05:21:58,173-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:22:08,181-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 05:22:12,958-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 05:22:18,189-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 05:22:25,878-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 05:22:28,197-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:22:38,205-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:22:48,213-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:22:48,567-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 05:22:58,201-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 05:22:58,222-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:23:08,230-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:23:18,237-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 05:23:24,177-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 05:23:28,246-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 05:23:30,523-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 05:23:38,254-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:23:48,262-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:23:58,270-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:23:59,787-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 05:24:02,848-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 05:24:08,278-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:24:18,286-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:24:28,294-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:24:35,171-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 05:24:35,396-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 05:24:38,302-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:24:48,310-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:24:58,318-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:25:07,494-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:25:08,326-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:25:11,007-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:25:18,334-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:25:28,342-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:25:38,350-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 05:25:39,818-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:25:46,617-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:25:48,358-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:25:58,366-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:26:08,374-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:26:12,140-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:26:18,382-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 05:26:22,229-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 05:26:28,390-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:26:38,399-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 05:26:44,464-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 05:26:48,406-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:26:57,837-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:26:58,414-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:27:08,422-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:27:16,786-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:27:18,430-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:27:28,438-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:27:33,447-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:27:38,446-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:27:48,454-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:27:49,109-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:27:58,462-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:28:08,470-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 05:28:09,057-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 05:28:18,478-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:28:21,432-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:28:28,487-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:28:38,495-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:28:44,667-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:28:48,502-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:28:53,755-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:28:58,510-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:29:08,519-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:29:18,526-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 05:29:20,277-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:29:26,078-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:29:28,534-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:29:38,542-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:29:48,551-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:29:55,888-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:29:58,401-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:29:58,559-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:30:08,567-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:30:18,575-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:30:28,583-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:30:30,724-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 7ms
-2013-11-06 05:30:31,504-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 7ms
-No streams
-2013-11-06 05:30:38,592-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:30:48,600-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:30:58,608-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:31:03,047-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:31:07,108-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:31:08,616-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:31:18,624-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:31:28,632-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:31:35,370-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:31:38,640-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:31:42,718-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:31:48,648-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:31:58,656-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 05:32:07,694-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 05:32:08,664-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 05:32:18,328-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 05:32:18,672-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:32:28,680-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:32:38,688-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:32:40,016-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:32:48,696-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 05:32:53,939-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 05:32:58,704-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:33:08,712-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:33:12,339-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:33:18,720-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:33:28,728-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:33:29,549-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:33:38,736-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:33:44,662-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:33:48,745-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:33:58,752-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 05:34:05,159-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 05:34:08,761-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:34:16,985-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:34:18,769-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:34:28,777-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:34:38,785-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:34:40,770-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:34:48,793-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:34:49,308-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:34:58,801-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:35:08,810-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 05:35:16,381-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 05:35:18,818-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:35:21,631-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:35:28,826-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:35:38,835-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:35:48,843-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:35:51,990-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:35:53,954-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:35:58,851-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:36:08,860-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:36:18,868-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:36:26,277-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:36:27,600-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:36:28,876-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:36:38,884-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:36:48,892-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:36:58,600-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:36:58,900-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 05:37:03,210-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 05:37:08,907-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:37:18,915-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:37:28,923-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:37:30,923-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:37:38,820-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:37:38,931-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:37:48,938-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:37:58,946-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:38:03,246-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:38:08,954-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:38:14,430-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:38:18,962-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:38:28,970-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 05:38:35,569-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 05:38:38,977-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:38:48,985-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 4ms
-2013-11-06 05:38:50,043-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 4ms
-No streams
-2013-11-06 05:38:58,993-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 05:39:07,892-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 05:39:09,001-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:39:19,008-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:39:25,650-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:39:29,016-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:39:39,024-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 05:39:40,216-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 05:39:49,033-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:39:59,041-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:40:01,260-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:40:09,048-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 05:40:12,538-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 05:40:19,056-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:40:29,064-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:40:36,870-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:40:39,072-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 05:40:44,861-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 05:40:49,080-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:40:59,087-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:41:09,095-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:41:12,480-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 05:41:17,184-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 05:41:19,103-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:41:29,111-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:41:39,119-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:41:48,090-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:41:49,127-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 05:41:49,507-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 05:41:59,135-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:42:09,143-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:42:19,151-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 05:42:21,831-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:42:23,700-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:42:29,159-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:42:39,167-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:42:49,175-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 05:42:54,155-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 05:42:59,183-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:42:59,310-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:43:09,191-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:43:19,199-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 05:43:26,478-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 05:43:29,207-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 05:43:34,920-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 05:43:39,215-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:43:49,223-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 05:43:58,802-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 05:43:59,231-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:44:09,239-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:44:10,530-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:44:19,247-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:44:29,255-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 05:44:31,125-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 05:44:39,262-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:44:46,140-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:44:49,271-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:44:59,279-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 05:45:03,448-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 05:45:09,287-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:45:19,294-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:45:21,750-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:45:29,302-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 05:45:35,771-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 05:45:39,311-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:45:49,319-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:45:57,360-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:45:59,328-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 05:46:08,094-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 05:46:09,336-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:46:19,344-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:46:29,352-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 05:46:32,970-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 05:46:39,360-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 05:46:40,417-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 05:46:49,368-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:46:59,376-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:47:08,580-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:47:09,385-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 05:47:12,740-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 05:47:19,393-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:47:29,401-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:47:39,409-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 3ms
-2013-11-06 05:47:44,193-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 3ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 05:47:45,064-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 05:47:49,417-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:47:59,425-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:48:09,433-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:48:17,386-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:48:19,441-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 05:48:19,802-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 05:48:29,449-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:48:39,458-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:48:49,466-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 05:48:49,710-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 05:48:55,412-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 05:48:59,473-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:49:09,481-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:49:19,490-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 3ms
-2013-11-06 05:49:22,034-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 3ms
-No streams
-2013-11-06 05:49:29,498-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 05:49:31,021-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 05:49:39,506-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:49:49,515-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 05:49:54,355-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 05:49:59,523-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:50:06,631-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:50:09,531-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:50:19,539-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 05:50:26,678-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 05:50:29,547-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:50:39,555-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:50:42,241-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:50:49,563-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 05:50:59,001-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 05:50:59,571-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:51:09,579-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:51:17,851-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:51:19,587-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:51:29,595-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 05:51:31,324-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 05:51:39,603-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:51:49,611-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:51:53,461-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:51:59,619-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 05:52:03,647-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 05:52:09,628-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:52:19,635-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 05:52:29,071-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 05:52:29,643-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 05:52:35,970-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 05:52:39,651-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:52:49,659-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:52:59,667-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:53:04,681-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 05:53:08,293-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 05:53:09,677-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:53:19,685-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:53:29,692-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:53:39,701-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:53:40,291-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 05:53:40,616-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 05:53:49,709-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:53:59,718-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:54:09,726-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:54:12,940-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 05:54:15,901-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 05:54:19,734-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:54:29,742-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:54:39,750-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 05:54:45,264-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 05:54:49,758-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:54:51,511-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:54:59,766-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:55:09,774-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 05:55:17,587-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 05:55:19,782-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:55:27,121-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:55:29,789-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:55:39,798-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:55:49,806-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 05:55:49,910-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 05:55:59,814-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:56:02,731-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:56:09,822-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:56:19,830-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 05:56:22,233-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 05:56:29,838-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:56:38,341-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:56:39,846-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:56:49,854-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 05:56:54,556-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 05:56:59,863-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:57:09,871-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 05:57:13,952-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 05:57:19,879-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 05:57:26,879-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 05:57:29,888-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:57:39,896-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 05:57:49,561-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 05:57:49,905-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 05:57:59,202-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 05:57:59,913-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:58:09,921-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:58:19,929-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:58:25,171-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:58:29,937-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 05:58:31,525-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 05:58:39,945-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:58:49,957-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:58:59,969-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:59:00,781-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 05:59:03,848-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 05:59:09,980-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:59:19,992-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:59:30,003-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:59:36,171-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 05:59:36,391-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 05:59:40,013-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 05:59:50,025-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:00:00,036-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:00:08,495-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:00:10,047-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:00:12,001-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 06:00:20,058-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:00:30,068-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:00:40,079-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:00:40,819-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:00:47,612-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 06:00:50,092-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:01:00,104-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:01:10,116-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:01:13,141-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 06:01:20,127-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:01:23,222-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 06:01:30,147-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:01:40,162-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:01:45,464-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:01:50,178-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:01:58,832-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 06:02:00,181-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:02:10,196-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:02:17,788-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 06:02:20,212-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:02:30,231-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:02:34,441-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 06:02:40,240-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:02:50,111-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:02:50,250-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:03:00,258-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:03:10,051-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:03:10,269-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:03:20,282-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:03:22,434-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 06:03:30,292-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:03:40,300-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:03:45,661-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:03:50,308-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:03:54,757-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 06:04:00,316-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:04:10,326-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:04:20,336-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:04:21,271-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:04:27,079-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 06:04:30,345-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:04:40,354-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:04:50,364-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:04:56,882-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:04:59,403-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 06:05:00,373-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:05:10,383-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:05:20,394-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:05:30,402-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:05:31,725-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-2013-11-06 06:05:32,493-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-No streams
-2013-11-06 06:05:40,411-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:05:50,421-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:06:00,431-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:06:04,049-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:06:08,101-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 06:06:10,442-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:06:20,451-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:06:30,452-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:06:36,372-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 06:06:40,465-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:06:43,711-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 06:06:50,475-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:07:00,485-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:07:08,695-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 06:07:10,493-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:07:19,322-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 06:07:20,503-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:07:30,505-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:07:40,510-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:07:41,018-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 06:07:50,530-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:07:54,932-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 06:08:00,534-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:08:10,542-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:08:13,341-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 06:08:20,553-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:08:30,541-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 06:08:30,561-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:08:40,569-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:08:45,663-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 06:08:50,578-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:09:00,587-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 06:09:06,152-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 06:09:10,595-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:09:17,987-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 06:09:20,604-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:09:30,612-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:09:40,622-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:09:41,763-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:09:50,310-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 06:09:50,630-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:10:00,639-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:10:10,648-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:10:17,373-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 06:10:20,657-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:10:22,633-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 06:10:30,666-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:10:40,679-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:10:50,688-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:10:52,983-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:10:54,956-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 06:11:00,699-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:11:10,710-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:11:20,719-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:11:27,278-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:11:28,593-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 06:11:30,728-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:11:40,738-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:11:50,748-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:11:59,602-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 06:12:00,758-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:12:04,202-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 06:12:10,768-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:12:20,777-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:12:30,788-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:12:31,925-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:12:39,812-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 06:12:40,799-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:12:50,808-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:13:00,817-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:13:04,247-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 06:13:10,827-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:13:15,423-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 06:13:20,835-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:13:30,844-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:13:36,571-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 06:13:40,852-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:13:50,862-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 06:13:51,033-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 06:14:00,872-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:14:08,894-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 06:14:10,882-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:14:20,893-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:14:26,642-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 06:14:30,909-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:14:40,923-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:14:41,216-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 06:14:50,934-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:15:00,942-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:15:02,252-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 06:15:10,951-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:15:13,540-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:15:20,961-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:15:30,970-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:15:37,863-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 06:15:40,979-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:15:45,864-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 06:15:50,988-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:16:00,997-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:16:11,004-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:16:13,472-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:16:18,187-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 06:16:21,013-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:16:31,022-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:16:41,032-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:16:49,083-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:16:50,509-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:16:51,043-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:17:01,051-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:17:11,060-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:17:21,069-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:17:22,833-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:17:24,693-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:17:31,078-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:17:41,086-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:17:51,094-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:17:55,156-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 06:18:00,303-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 06:18:01,102-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:18:11,109-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:18:21,118-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:18:27,479-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:18:31,125-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:18:35,912-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:18:41,134-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:18:51,142-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:18:59,801-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 06:19:01,153-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:19:11,162-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:19:11,522-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 06:19:21,172-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:19:31,182-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:19:32,124-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 06:19:41,194-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:19:47,133-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 06:19:51,205-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:20:01,215-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:20:04,448-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 06:20:11,223-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:20:21,232-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:20:22,742-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 06:20:31,242-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:20:36,770-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 06:20:41,252-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:20:51,262-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:20:58,353-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 06:21:01,272-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:21:09,093-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 06:21:11,280-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:21:21,289-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:21:31,298-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:21:33,963-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 06:21:41,306-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:21:41,416-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:21:51,315-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:22:01,324-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:22:09,572-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 06:22:11,334-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:22:13,740-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:22:21,344-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:22:31,355-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:22:41,364-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:22:45,183-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:22:46,064-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:22:51,373-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:23:01,383-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:23:11,393-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:23:18,387-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:23:20,792-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 06:23:21,403-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:23:31,413-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:23:41,422-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:23:50,710-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 06:23:51,432-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:23:56,403-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 06:24:01,441-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:24:11,449-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:24:21,458-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:24:23,033-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 06:24:31,468-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:24:32,013-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 06:24:41,477-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:24:51,487-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:24:55,357-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 06:25:01,496-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:25:07,623-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 06:25:11,505-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:25:21,514-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:25:27,679-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 06:25:31,523-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:25:41,533-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:25:43,233-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 06:25:51,541-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:26:00,003-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 06:26:01,551-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:26:11,560-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:26:18,843-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 06:26:21,568-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:26:31,576-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:26:32,325-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 06:26:41,586-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:26:51,595-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:26:54,453-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 06:27:01,603-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:27:04,648-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 06:27:11,612-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:27:21,621-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:27:30,063-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:27:31,631-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:27:36,972-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 06:27:41,639-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:27:51,647-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:28:01,656-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:28:05,673-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:28:09,295-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 06:28:11,665-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:28:21,673-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:28:31,682-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:28:41,284-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:28:41,618-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 06:28:41,690-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:28:51,697-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:29:01,706-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:29:11,714-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:29:13,941-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:29:16,893-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:29:21,722-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:29:31,729-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:29:41,738-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:29:46,264-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:29:51,745-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:29:52,503-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 06:30:01,753-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:30:11,761-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:30:18,587-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:30:21,769-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:30:28,113-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 06:30:31,777-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:30:41,786-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:30:50,910-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:30:51,794-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:31:01,802-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:31:03,724-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 06:31:11,810-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:31:21,818-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:31:23,233-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:31:31,827-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:31:39,333-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:31:41,834-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:31:51,843-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:31:55,556-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:32:01,851-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:32:11,859-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:32:14,944-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 06:32:21,867-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:32:27,879-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:32:31,876-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:32:41,884-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 06:32:50,554-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 06:32:51,892-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:33:00,202-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:33:01,900-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:33:11,908-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:33:21,916-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:33:26,164-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:33:31,924-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:33:32,525-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:33:41,932-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:33:51,940-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:34:01,774-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 06:34:01,949-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:34:04,848-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:34:11,957-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:34:21,965-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:34:31,974-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:34:37,171-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:34:37,384-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:34:41,982-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:34:51,992-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:35:02,000-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:35:09,494-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:35:12,008-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:35:12,994-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 06:35:22,017-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:35:32,025-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:35:41,817-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:35:42,035-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:35:48,604-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:35:52,043-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:36:02,051-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:36:12,060-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 4ms
-2013-11-06 06:36:14,143-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 4ms
-No streams
-2013-11-06 06:36:22,068-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:36:24,215-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 06:36:32,076-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:36:42,084-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:36:46,463-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:36:52,093-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:36:59,824-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 06:37:02,101-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:37:12,109-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:37:18,786-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:37:22,118-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:37:32,126-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:37:35,434-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:37:42,135-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:37:51,109-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:37:52,143-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:38:02,151-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:38:11,044-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 06:38:12,159-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:38:22,167-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:38:23,432-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 06:38:32,175-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:38:42,183-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:38:46,654-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:38:52,191-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:38:55,755-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:39:02,200-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:39:12,208-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:39:22,217-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:39:22,264-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:39:28,078-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:39:32,225-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:39:42,234-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:39:52,242-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:39:57,875-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:40:00,401-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:40:02,250-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:40:12,258-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:40:22,267-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:40:32,275-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:40:32,724-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 06:40:33,485-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 06:40:42,283-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:40:52,292-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:41:02,300-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:41:05,047-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:41:09,094-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:41:12,308-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:41:22,316-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:41:32,328-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:41:37,370-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:41:42,336-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:41:44,704-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:41:52,344-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:42:02,352-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:42:09,694-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 06:42:12,361-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:42:20,314-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 06:42:22,369-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:42:32,377-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:42:42,016-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:42:42,385-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:42:52,393-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:42:55,924-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:43:02,401-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:43:12,410-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:43:14,340-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 06:43:22,418-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:43:31,534-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 06:43:32,426-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:43:42,434-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:43:46,662-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 06:43:52,442-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:44:02,450-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:44:07,144-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:44:12,459-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:44:18,986-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 06:44:22,467-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:44:32,475-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:44:42,483-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:44:42,755-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:44:51,309-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 06:44:52,494-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:45:02,502-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:45:12,510-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:45:18,365-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:45:22,519-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:45:23,632-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 06:45:32,527-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:45:42,536-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:45:52,544-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:45:53,975-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:45:55,954-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 06:46:02,553-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:46:12,561-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:46:22,569-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:46:28,278-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:46:29,586-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:46:32,579-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:46:42,587-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:46:52,595-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:47:00,600-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 06:47:02,603-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:47:05,196-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:47:12,611-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:47:22,620-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:47:32,629-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:47:32,923-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:47:40,806-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 06:47:42,637-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:47:52,645-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:48:02,653-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:48:05,247-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 06:48:12,661-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:48:16,416-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:48:22,669-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:48:32,678-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:48:37,569-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 06:48:42,686-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:48:52,027-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 06:48:52,694-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:49:02,702-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:49:09,892-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 06:49:12,710-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:49:22,718-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:49:27,636-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:49:32,726-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:49:42,215-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:49:42,734-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:49:52,742-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:50:02,750-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:50:03,246-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:50:12,759-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:50:14,539-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 06:50:22,767-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:50:32,775-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:50:38,857-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 06:50:42,783-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:50:46,861-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:50:52,792-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:51:02,800-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:51:12,808-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:51:14,466-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:51:19,184-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:51:22,816-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:51:32,824-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:51:42,833-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:51:50,076-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:51:51,507-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:51:52,841-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:52:02,849-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:52:12,858-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:52:22,866-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:52:23,830-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:52:25,686-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:52:32,874-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:52:42,883-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:52:52,891-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:52:56,153-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:53:01,297-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 06:53:02,899-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:53:12,907-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:53:22,916-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:53:28,476-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 06:53:32,924-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:53:36,906-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 06:53:42,932-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:53:52,940-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:54:00,799-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:54:02,949-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:54:12,516-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:54:12,957-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:54:22,965-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:54:32,973-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:54:33,122-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:54:42,981-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:54:48,126-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:54:52,989-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:55:02,997-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:55:05,446-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 06:55:13,006-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:55:23,014-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:55:23,737-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:55:33,022-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:55:37,769-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 06:55:43,031-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:55:53,039-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:55:59,347-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 06:56:03,047-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:56:10,092-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 06:56:13,056-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:56:23,064-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:56:33,072-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:56:34,957-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-06 06:56:42,414-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-No streams
-2013-11-06 06:56:43,081-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:56:53,089-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:57:03,097-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 17ms
-2013-11-06 06:57:10,583-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 17ms
-No streams
-2013-11-06 06:57:13,106-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:57:14,738-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 06:57:23,114-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:57:33,122-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:57:43,130-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:57:46,177-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 06:57:47,062-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 06:57:53,139-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:58:03,147-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:58:13,156-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:58:19,384-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:58:21,787-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:58:23,164-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:58:33,172-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:58:43,180-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:58:51,707-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:58:53,189-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:58:57,397-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 06:59:03,197-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:59:13,205-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:59:23,214-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 06:59:24,030-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:59:33,007-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 06:59:33,222-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:59:43,230-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 06:59:53,239-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 06:59:56,353-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 07:00:03,247-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:00:08,617-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:00:13,255-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:00:23,263-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 07:00:28,677-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 07:00:33,272-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:00:43,280-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 07:00:44,228-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 07:00:53,288-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 07:01:01,000-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 07:01:03,297-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:01:13,305-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 07:01:19,838-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 07:01:23,313-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:01:33,321-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:01:33,322-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 07:01:43,330-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:01:53,338-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:01:55,447-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:02:03,346-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 07:02:05,646-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 07:02:13,355-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:02:23,363-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:02:31,057-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 07:02:33,371-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:02:37,968-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 07:02:43,379-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:02:53,387-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:03:03,396-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:03:06,667-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:03:10,292-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:03:13,404-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:03:23,412-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:03:33,420-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:03:42,277-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:03:42,614-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:03:43,428-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:03:53,436-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:04:03,445-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:04:13,453-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:04:14,937-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:04:17,887-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:04:23,461-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:04:33,469-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:04:43,477-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:04:47,260-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 07:04:53,485-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:04:53,497-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:05:03,493-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:05:13,502-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 07:05:19,584-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 07:05:23,510-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:05:29,107-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:05:33,519-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:05:43,527-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 07:05:51,907-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 07:05:53,536-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:06:03,544-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:06:04,717-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:06:13,552-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:06:23,560-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 07:06:24,230-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 07:06:33,568-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:06:40,327-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 07:06:43,577-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:06:53,585-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 07:06:56,553-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 07:07:03,593-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:07:13,602-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:07:15,937-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:07:23,610-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 07:07:28,876-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 07:07:33,618-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:07:43,626-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:07:51,547-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 07:07:53,635-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 07:08:01,199-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 07:08:03,643-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:08:13,651-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:08:23,660-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 07:08:27,157-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:08:33,521-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 07:08:33,668-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:08:43,676-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:08:53,684-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:09:02,768-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:09:03,693-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 07:09:05,845-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 07:09:13,701-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:09:23,709-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:09:33,717-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:09:38,167-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 07:09:38,378-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 07:09:43,726-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:09:53,734-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:10:03,743-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:10:10,490-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 07:10:13,751-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:10:13,989-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:10:23,759-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:10:33,767-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 07:10:42,814-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 07:10:43,775-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:10:49,599-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 07:10:53,783-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:11:03,791-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:11:13,800-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:11:15,137-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 07:11:23,808-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:11:25,209-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:11:33,816-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:11:43,825-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:11:47,459-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:11:53,833-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:12:00,819-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:12:03,841-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:12:13,849-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:12:19,782-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 07:12:23,858-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:12:33,866-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:12:36,429-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 07:12:43,874-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:12:52,105-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 07:12:53,883-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:13:03,891-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:13:12,039-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:13:13,899-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:13:23,907-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:13:24,428-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 07:13:33,916-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:13:43,924-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:13:47,649-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:13:53,932-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:13:56,751-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 07:14:03,941-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:14:13,949-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:14:23,259-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:14:23,957-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:14:29,074-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 07:14:33,965-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:14:43,974-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:14:53,982-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:14:58,869-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 07:15:01,398-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 07:15:03,990-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:15:13,998-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:15:24,006-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 07:15:33,721-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 07:15:34,015-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:15:34,480-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 07:15:44,023-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:15:54,031-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:16:04,039-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:16:06,043-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:16:10,089-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:16:14,048-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:16:24,056-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:16:34,064-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:16:38,366-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 07:16:44,072-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:16:45,699-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 07:16:54,080-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:17:04,088-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 07:17:10,690-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 07:17:14,097-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:17:21,309-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:17:24,105-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:17:34,112-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 07:17:43,013-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 07:17:44,121-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:17:54,128-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:17:56,919-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 07:18:04,136-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:18:14,145-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:18:15,336-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:18:24,153-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:18:32,529-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:18:34,162-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:18:44,170-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:18:47,660-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:18:54,178-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:19:04,186-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:19:08,139-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:19:14,195-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:19:19,984-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:19:24,203-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:19:34,211-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 07:19:43,751-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 07:19:44,220-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 07:19:52,306-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 07:19:54,228-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:20:04,236-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:20:14,244-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:20:19,359-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:20:24,252-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:20:24,630-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:20:34,261-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:20:44,269-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:20:54,277-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:20:54,969-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:20:56,953-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:21:04,285-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:21:14,293-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:21:24,301-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:21:29,277-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:21:30,579-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:21:34,310-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:21:44,318-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:21:54,326-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 07:22:01,601-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 07:22:04,334-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:22:06,189-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 07:22:14,342-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:22:24,351-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 07:22:33,924-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 07:22:34,358-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 07:22:41,799-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 07:22:44,366-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:22:54,375-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:23:04,383-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:23:06,247-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:23:14,391-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:23:17,409-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:23:24,399-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:23:34,408-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 07:23:38,571-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 07:23:44,416-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 07:23:53,020-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 07:23:54,424-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:24:04,432-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:24:10,893-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 07:24:14,440-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:24:24,448-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:24:28,629-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:24:34,456-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 07:24:43,217-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 07:24:44,465-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:24:54,473-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:25:04,239-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:25:04,480-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:25:14,488-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 07:25:15,540-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 07:25:24,497-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:25:34,505-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 07:25:39,850-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 07:25:44,513-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 07:25:47,863-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 07:25:54,521-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:26:04,529-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:26:14,537-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:26:15,459-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:26:20,186-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:26:24,545-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:26:34,554-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:26:44,562-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:26:51,069-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:26:52,509-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:26:54,570-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:27:04,578-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:27:14,586-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:27:24,594-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:27:24,832-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:27:26,679-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:27:34,602-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:27:44,610-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:27:54,618-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:27:57,155-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 07:28:02,290-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 07:28:04,626-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:28:14,633-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:28:24,642-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:28:29,478-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 07:28:34,649-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:28:37,899-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 07:28:44,657-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:28:54,665-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:29:01,801-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:29:04,673-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:29:13,509-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:29:14,681-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:29:24,689-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:29:34,124-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 07:29:34,697-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:29:44,705-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:29:49,119-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:29:54,713-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:30:04,721-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 07:30:06,448-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 07:30:14,729-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:30:24,729-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:30:24,737-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:30:34,745-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 6ms
-2013-11-06 07:30:38,775-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 6ms
-No streams
-2013-11-06 07:30:44,753-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:30:54,762-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:31:00,339-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:31:04,770-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:31:11,094-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 07:31:14,778-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:31:24,786-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:31:34,794-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:31:35,949-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:31:43,416-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 07:31:44,802-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:31:54,810-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:32:04,818-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 07:32:11,560-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 07:32:14,827-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:32:15,739-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 07:32:24,835-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:32:34,843-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:32:44,851-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:32:47,169-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:32:48,062-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 07:32:54,860-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:33:04,868-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:33:14,876-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:33:20,385-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 07:33:22,778-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 07:33:24,885-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:33:34,893-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:33:44,902-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:33:52,708-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 07:33:54,910-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:33:58,389-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:34:04,918-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:34:14,927-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:34:24,935-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:34:25,031-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:34:33,999-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 07:34:34,943-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:34:44,951-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:34:54,960-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:34:57,354-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:35:04,968-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:35:09,609-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:35:14,976-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:35:24,984-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 07:35:29,678-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 07:35:34,993-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:35:45,001-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 07:35:45,220-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 07:35:55,009-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 07:36:02,001-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 07:36:05,018-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:36:15,026-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 07:36:20,830-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 07:36:25,035-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:36:34,323-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:36:35,043-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:36:45,051-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:36:55,059-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:36:56,440-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:37:05,068-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:37:06,646-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:37:15,076-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:37:25,084-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:37:32,049-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:37:35,092-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:37:38,969-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:37:45,100-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:37:55,108-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:38:05,116-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:38:07,659-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 17ms
-2013-11-06 07:38:11,308-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 17ms
-No streams
-2013-11-06 07:38:15,124-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:38:25,132-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:38:35,141-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:38:43,269-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:38:43,615-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:38:45,149-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:38:55,157-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:39:05,166-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:39:15,174-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:39:15,938-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:39:18,879-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 07:39:25,182-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:39:35,191-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:39:45,199-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:39:48,261-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:39:54,489-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:39:55,207-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:40:05,216-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:40:15,225-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 07:40:20,585-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 07:40:25,235-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 07:40:30,100-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 07:40:35,243-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:40:45,251-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:40:52,907-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:40:55,260-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:41:05,268-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:41:05,709-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 07:41:15,276-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:41:25,230-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:41:25,284-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:41:35,292-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 3ms
-2013-11-06 07:41:41,321-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 3ms
-No streams
-2013-11-06 07:41:45,301-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:41:55,309-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:41:57,553-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:42:05,317-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:42:15,325-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:42:16,929-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:42:25,333-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:42:29,876-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:42:35,341-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:42:45,349-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:42:52,539-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:42:55,357-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 07:43:02,200-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 07:43:05,365-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:43:15,373-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:43:25,381-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:43:28,149-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:43:34,522-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:43:35,389-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:43:45,397-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:43:55,406-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:44:03,759-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 07:44:05,414-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:44:06,845-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:44:15,422-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:44:25,431-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:44:35,439-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:44:39,168-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 07:44:39,370-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 07:44:45,447-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:44:55,456-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:45:05,464-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:45:11,491-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:45:14,979-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 07:45:15,472-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:45:25,480-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:45:35,489-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:45:43,814-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:45:45,497-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:45:50,589-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:45:55,505-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:46:05,513-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:46:15,521-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:46:16,137-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:46:25,529-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:46:26,199-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:46:35,538-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:46:45,546-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:46:48,460-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:46:55,554-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:47:01,809-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 07:47:05,562-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:47:15,570-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:47:20,783-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:47:25,578-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:47:35,586-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:47:37,419-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:47:45,594-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:47:53,106-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:47:55,602-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:48:05,610-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 07:48:13,030-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 07:48:15,618-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:48:25,429-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:48:25,626-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:48:35,634-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:48:45,641-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:48:48,639-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 07:48:55,649-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:48:57,752-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:49:05,657-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:49:15,665-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:49:24,249-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:49:25,673-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:49:30,075-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:49:35,681-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:49:45,688-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:49:55,696-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:49:59,859-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 07:50:02,397-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 07:50:05,704-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:50:15,712-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:50:25,720-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 07:50:34,722-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:50:35,469-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:50:35,728-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:50:45,735-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:50:55,743-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:51:05,751-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 07:51:07,045-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:51:11,079-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:51:15,759-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:51:25,767-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:51:35,775-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:51:39,367-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 07:51:45,783-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 5ms
-2013-11-06 07:51:46,693-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 5ms
-No streams
-2013-11-06 07:51:55,791-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:52:05,799-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 07:52:11,691-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 07:52:15,807-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:52:22,299-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:52:25,815-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:52:35,823-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:52:44,013-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 07:52:45,831-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:52:55,839-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:52:57,909-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:53:05,846-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:53:15,854-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:53:16,336-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 07:53:25,862-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 07:53:33,519-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 07:53:35,870-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:53:45,878-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:53:48,659-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 07:53:55,886-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:54:05,893-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:54:09,130-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:54:15,901-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:54:20,982-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 07:54:25,909-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:54:35,917-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 07:54:44,741-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 07:54:45,925-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:54:53,305-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 07:54:55,933-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:55:05,940-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:55:15,948-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:55:20,351-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 07:55:25,629-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 07:55:25,956-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:55:35,964-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:55:45,972-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:55:55,961-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:55:55,979-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:55:57,951-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 07:56:05,987-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:56:15,995-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:56:26,003-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:56:30,274-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 07:56:31,571-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 07:56:36,011-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:56:46,019-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:56:56,027-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:57:02,597-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:57:06,034-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:57:07,182-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:57:16,042-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:57:26,050-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:57:34,920-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 07:57:36,058-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:57:42,792-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 07:57:46,066-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:57:56,074-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:58:06,082-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:58:07,243-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 07:58:16,090-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 07:58:18,402-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 07:58:26,098-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:58:36,106-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:58:39,566-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:58:46,115-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 07:58:54,014-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 07:58:56,123-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:59:06,131-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:59:11,889-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:59:16,140-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:59:26,148-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 07:59:29,623-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 07:59:36,156-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 07:59:44,212-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 07:59:46,165-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 07:59:56,173-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 08:00:05,234-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 08:00:06,181-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:00:16,189-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 08:00:16,535-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 08:00:26,197-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:00:36,205-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 08:00:40,844-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 08:00:46,213-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 08:00:48,858-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 08:00:56,221-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:01:06,229-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:01:16,238-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:01:16,454-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 08:01:21,181-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 08:01:26,246-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:01:36,254-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:01:46,262-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 4ms
-2013-11-06 08:01:52,067-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 4ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 08:01:53,504-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 08:01:56,271-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:02:06,279-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:02:16,287-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 08:02:25,827-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 08:02:26,295-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 08:02:27,674-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 08:02:36,303-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:02:46,311-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:02:56,319-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 08:02:58,150-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 08:03:03,285-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 08:03:06,327-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:03:16,335-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:03:26,343-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 08:03:30,474-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 08:03:36,351-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 08:03:38,893-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 08:03:46,359-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:03:56,367-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 08:04:02,798-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 08:04:06,375-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 08:04:14,504-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 08:04:16,383-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:04:26,391-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:04:35,122-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:04:36,399-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:04:46,407-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:04:50,114-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:04:56,415-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:05:06,423-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:05:07,444-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:05:16,430-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:05:25,724-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:05:26,439-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:05:36,446-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:05:39,768-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:05:46,455-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:05:56,462-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:06:01,334-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:06:06,470-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 08:06:12,091-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 08:06:16,478-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:06:26,486-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:06:36,494-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:06:36,944-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:06:44,413-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:06:46,502-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:06:56,510-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:07:06,518-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 08:07:12,555-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 08:07:16,526-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:07:16,736-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:07:26,534-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:07:36,542-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:07:46,550-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 08:07:48,164-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:07:49,059-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:07:56,558-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:08:06,566-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:08:16,573-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:08:21,382-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 08:08:23,775-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 08:08:26,582-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:08:36,590-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:08:46,598-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:08:53,705-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:08:56,606-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 08:08:59,385-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 08:09:06,614-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:09:16,622-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:09:26,028-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:09:26,630-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:09:34,995-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:09:36,638-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:09:46,646-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:09:56,654-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:09:58,351-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:10:06,662-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 08:10:10,606-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 08:10:16,670-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:10:26,679-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:10:30,674-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:10:36,687-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 08:10:46,216-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 08:10:46,695-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:10:56,703-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:11:02,997-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:11:06,711-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:11:16,719-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 08:11:21,827-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 08:11:26,727-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:11:35,320-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:11:36,735-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:11:46,743-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:11:56,751-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 3ms
-2013-11-06 08:11:57,439-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 3ms
-No streams
-2013-11-06 08:12:06,760-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 08:12:07,643-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 08:12:16,768-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:12:26,776-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 08:12:33,047-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 08:12:36,784-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 08:12:39,966-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 08:12:46,792-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:12:56,800-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:13:06,808-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:13:08,657-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 08:13:12,289-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 08:13:16,816-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:13:26,824-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:13:36,832-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 08:13:44,267-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 08:13:44,612-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 08:13:46,840-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:13:56,848-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:14:06,856-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:14:16,864-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 08:14:16,936-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:14:19,877-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:14:26,872-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:14:36,880-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:14:46,888-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 08:14:49,259-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:14:55,487-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:14:56,896-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:15:06,904-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:15:16,912-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 08:15:21,583-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 08:15:26,920-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 08:15:31,098-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 08:15:36,928-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:15:46,936-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 08:15:53,906-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 08:15:56,944-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:16:06,707-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:16:06,952-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:16:16,960-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 08:16:26,229-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 08:16:26,968-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:16:36,976-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 08:16:42,317-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 08:16:46,984-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:16:56,992-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 08:16:58,552-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 08:17:07,000-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:17:17,008-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:17:17,927-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:17:27,016-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 08:17:30,876-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 08:17:37,024-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:17:47,032-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:17:53,537-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:17:57,040-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 08:18:03,200-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 08:18:07,048-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:18:17,056-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:18:27,064-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 08:18:29,147-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 08:18:35,524-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 08:18:37,072-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:18:47,080-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:18:57,088-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:19:04,757-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:19:07,096-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:19:07,847-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:19:17,104-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:19:27,112-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:19:37,120-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-2013-11-06 08:19:40,172-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 08:19:40,368-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 08:19:47,128-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:19:57,136-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:20:07,144-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:20:12,493-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:20:15,977-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:20:17,152-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:20:27,160-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:20:37,168-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 08:20:44,817-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 08:20:47,176-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:20:51,587-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:20:57,184-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:21:07,192-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:21:17,139-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:21:17,200-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 08:21:27,196-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 08:21:27,208-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:21:37,216-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:21:47,224-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:21:49,462-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:21:57,232-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 3ms
-2013-11-06 08:22:02,809-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 3ms
-No streams
-2013-11-06 08:22:07,240-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:22:17,249-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:22:21,785-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:22:27,257-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:22:37,265-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:22:38,417-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:22:47,273-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:22:54,108-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:22:57,282-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:23:07,290-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:23:14,027-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:23:17,298-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:23:26,431-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:23:27,307-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:23:37,315-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:23:47,323-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 5ms
-2013-11-06 08:23:49,641-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 5ms
-No streams
-2013-11-06 08:23:57,331-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:23:58,754-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:24:07,340-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:24:17,348-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:24:25,247-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:24:27,356-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:24:31,077-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:24:37,364-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:24:47,373-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:24:57,380-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:25:00,857-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:25:03,400-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:25:07,389-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:25:17,397-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:25:27,405-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:25:35,723-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:25:36,467-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:25:37,413-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:25:47,421-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:25:57,429-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:26:07,437-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:26:08,046-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:26:12,077-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:26:17,445-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:26:27,453-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:26:37,462-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:26:40,369-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:26:47,469-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 08:26:47,687-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 08:26:57,477-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:27:07,485-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 08:27:12,693-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 08:27:17,493-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:27:23,297-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:27:27,501-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:27:37,510-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:27:45,015-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:27:47,518-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:27:57,525-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 08:27:58,908-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 08:28:07,533-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 08:28:17,338-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 08:28:17,542-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:28:27,549-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 08:28:34,517-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 08:28:37,558-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:28:47,565-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:28:49,661-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:28:57,573-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:29:07,581-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:29:10,127-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:29:17,589-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 08:29:21,985-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 08:29:27,597-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:29:37,605-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:29:45,737-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:29:47,613-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:29:54,308-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:29:57,621-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:30:07,629-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:30:17,637-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:30:21,347-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:30:26,632-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:30:27,645-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:30:37,653-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:30:47,661-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:30:56,957-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:30:57,670-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 08:30:58,955-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 08:31:07,678-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:31:17,686-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:31:27,694-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:31:31,278-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:31:32,567-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:31:37,703-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:31:47,711-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:31:57,719-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:32:03,601-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:32:07,728-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 5ms
-2013-11-06 08:32:08,181-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 5ms
-No streams
-2013-11-06 08:32:17,730-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:32:27,738-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:32:35,924-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:32:37,746-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:32:43,787-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:32:47,754-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:32:57,762-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:33:07,770-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 114ms
-2013-11-06 08:33:08,360-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 114ms
-No streams
-2013-11-06 08:33:17,778-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 08:33:19,397-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 08:33:27,786-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:33:37,794-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:33:40,570-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:33:47,802-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 08:33:55,008-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 08:33:57,810-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:34:07,818-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:34:12,893-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:34:17,826-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:34:27,834-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:34:30,617-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:34:37,842-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:34:45,216-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:34:47,850-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:34:57,858-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 08:35:06,228-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 08:35:07,867-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:35:17,539-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:35:17,875-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:35:27,883-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:35:37,891-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 08:35:41,839-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 08:35:47,899-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:35:49,862-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:35:57,907-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:36:07,915-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 08:36:17,449-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 08:36:17,924-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 08:36:22,186-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 08:36:27,932-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:36:37,940-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:36:47,948-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:36:53,058-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:36:54,508-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:36:57,956-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:37:07,964-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:37:17,973-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:37:26,831-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:37:27,980-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:37:28,668-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:37:37,989-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:37:47,997-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:37:58,005-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:37:59,154-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 08:38:04,279-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 08:38:08,014-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:38:18,022-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:38:28,030-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:38:31,477-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:38:38,038-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:38:39,888-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:38:48,046-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:38:58,055-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:39:03,800-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:39:08,063-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:39:15,498-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:39:18,071-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:39:28,079-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:39:36,123-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:39:38,087-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:39:48,095-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 08:39:51,108-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 08:39:58,104-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:40:08,113-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:40:08,446-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:40:18,122-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 08:40:26,720-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 08:40:28,130-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:40:38,138-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:40:40,769-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:40:48,146-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:40:58,154-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:41:02,329-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:41:08,162-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:41:13,092-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:41:18,170-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:41:28,178-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:41:37,939-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:41:38,187-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:41:45,415-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:41:48,194-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:41:58,202-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:42:08,210-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 5ms
-2013-11-06 08:42:13,553-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 5ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:42:17,738-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:42:18,219-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:42:28,226-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:42:38,235-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:42:48,243-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:42:49,159-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:42:50,061-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:42:58,251-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:43:08,259-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:43:18,267-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:43:22,384-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:43:24,769-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:43:28,275-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:43:38,283-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:43:48,291-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:43:54,707-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:43:58,298-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:44:00,379-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:44:08,306-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:44:18,314-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:44:27,030-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:44:28,322-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:44:35,989-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:44:38,330-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:44:48,338-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:44:58,346-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:44:59,353-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:45:08,354-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 08:45:11,600-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 08:45:18,362-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:45:28,371-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 08:45:31,677-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 08:45:38,379-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:45:47,209-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:45:48,386-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:45:58,394-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:46:03,999-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:46:08,402-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:46:18,410-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 08:46:22,820-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 08:46:28,418-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:46:36,322-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:46:38,425-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:46:48,433-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 08:46:58,429-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 08:46:58,441-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:47:08,449-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:47:08,645-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:47:18,457-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:47:28,464-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:47:34,039-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:47:38,472-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:47:40,968-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:47:48,481-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:47:58,488-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:48:08,497-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:48:09,649-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:48:13,291-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:48:18,505-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:48:28,513-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:48:38,521-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:48:45,259-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:48:45,614-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:48:48,530-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:48:58,538-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:49:08,548-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:49:17,937-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:49:18,556-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:49:20,869-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:49:28,564-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:49:38,572-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:49:48,581-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:49:50,260-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 08:49:56,478-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 08:49:58,592-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:50:08,600-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:50:18,608-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:50:22,583-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:50:28,616-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 08:50:32,090-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 08:50:38,624-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:50:48,632-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:50:54,906-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:50:58,641-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:51:07,699-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:51:08,649-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:51:18,657-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:51:27,229-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:51:28,665-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:51:38,673-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:51:43,309-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:51:48,682-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:51:58,690-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:51:59,552-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:52:08,698-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:52:18,706-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 4ms
-2013-11-06 08:52:18,922-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 4ms
-No streams
-2013-11-06 08:52:28,714-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:52:31,875-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:52:38,723-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:52:48,731-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 08:52:54,530-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 08:52:58,739-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 08:53:04,198-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 08:53:08,748-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:53:18,756-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:53:28,765-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:53:30,140-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:53:36,521-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:53:38,773-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:53:48,782-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:53:58,790-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:54:05,750-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:54:08,798-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:54:08,844-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:54:18,806-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:54:28,814-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:54:38,822-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:54:41,167-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:54:41,360-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:54:48,830-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:54:58,838-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:55:08,845-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:55:13,490-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:55:16,970-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:55:18,853-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:55:28,861-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:55:38,870-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:55:45,813-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:55:48,877-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:55:52,580-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:55:58,886-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:56:08,894-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:56:18,136-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:56:18,902-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:56:28,190-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:56:28,910-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:56:38,918-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:56:48,926-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:56:50,459-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:56:58,934-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-06 08:57:03,799-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-06 08:57:08,943-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:57:18,951-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:57:22,782-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:57:28,959-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:57:38,967-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:57:39,410-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:57:48,976-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:57:55,105-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:57:58,984-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:58:08,992-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:58:15,020-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:58:19,001-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:58:27,428-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:58:29,009-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:58:39,017-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:58:49,025-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:58:50,630-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:58:59,034-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:58:59,751-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:59:09,042-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:59:19,050-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:59:26,240-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:59:29,058-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 08:59:32,074-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 08:59:39,067-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:59:49,075-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 08:59:59,083-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:00:01,850-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:00:04,397-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 09:00:09,091-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:00:19,099-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:00:29,108-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:00:36,720-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:00:37,460-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:00:39,116-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:00:49,124-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:00:59,133-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:01:09,043-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 09:01:09,141-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:01:13,070-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:01:19,149-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:01:29,157-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:01:39,165-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:01:41,366-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:01:48,680-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:01:49,173-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:01:59,181-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:02:09,190-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:02:13,689-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 09:02:19,198-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 3ms
-2013-11-06 09:02:24,292-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 3ms
-No streams
-2013-11-06 09:02:29,206-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:02:39,214-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:02:46,012-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 09:02:49,222-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:02:59,230-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:02:59,900-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:03:09,238-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:03:18,335-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:03:19,247-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:03:29,255-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:03:35,510-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:03:39,263-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:03:49,271-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:03:50,658-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:03:59,279-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:04:09,287-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:04:11,120-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:04:19,295-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:04:22,981-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:04:29,303-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:04:39,311-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:04:46,730-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:04:49,319-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:04:55,304-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:04:59,327-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:05:09,335-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:05:19,344-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:05:22,340-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:05:27,627-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:05:29,352-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:05:39,360-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:05:49,369-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:05:57,951-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:05:59,377-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:05:59,950-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:06:09,385-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:06:19,393-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:06:29,401-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:06:32,273-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:06:33,561-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:06:39,410-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:06:49,418-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:06:59,426-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:07:04,596-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:07:09,171-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:07:09,435-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:07:19,443-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:07:29,451-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:07:36,919-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:07:39,460-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:07:44,781-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:07:49,468-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:07:59,476-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:08:09,242-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:08:09,484-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:08:19,492-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:08:20,391-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:08:29,501-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:08:39,509-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:08:41,565-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:08:49,517-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:08:56,001-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:08:59,525-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:09:09,534-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:09:13,888-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:09:19,543-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:09:29,551-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:09:31,611-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:09:39,559-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:09:46,211-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:09:49,567-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:09:59,575-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:10:07,221-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:10:09,584-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:10:18,534-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 09:10:19,592-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:10:29,600-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:10:39,608-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:10:42,831-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:10:49,616-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:10:50,857-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 09:10:59,624-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:11:09,632-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:11:18,441-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:11:19,640-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 09:11:23,181-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 09:11:29,649-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:11:39,657-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:11:49,665-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-06 09:11:54,050-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:11:55,503-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:11:59,673-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:12:09,681-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:12:19,690-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:12:27,826-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 3ms
-2013-11-06 09:12:29,663-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 3ms
-No streams
-2013-11-06 09:12:29,698-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:12:39,706-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:12:49,715-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:12:59,723-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:13:00,149-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:13:05,272-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:13:09,731-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:13:19,739-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:13:29,747-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:13:32,472-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:13:39,755-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:13:40,882-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:13:49,764-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:13:59,772-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:14:04,795-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:14:09,780-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:14:16,492-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:14:19,789-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:14:29,797-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:14:37,118-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:14:39,806-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:14:49,814-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-06 09:14:52,101-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-06 09:14:59,822-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:15:09,441-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:15:09,831-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:15:19,839-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:15:27,713-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:15:29,848-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:15:39,856-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:15:41,764-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 09:15:49,865-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:15:59,873-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:16:03,322-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:16:09,882-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:16:14,087-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:16:19,890-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:16:29,898-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:16:38,932-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:16:39,906-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:16:46,410-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 09:16:49,914-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:16:59,922-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:17:09,930-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 3ms
-2013-11-06 09:17:14,544-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 3ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:17:18,733-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 09:17:19,938-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:17:29,946-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:17:39,954-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:17:49,962-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:17:50,152-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:17:51,056-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 09:17:59,970-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:18:09,977-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:18:19,985-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:18:23,379-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:18:25,762-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:18:29,993-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:18:40,001-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:18:50,009-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:18:55,702-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 09:19:00,017-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:19:01,372-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:19:10,025-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:19:20,033-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:19:28,025-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:19:30,040-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:19:36,982-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:19:40,048-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:19:50,056-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:20:00,064-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:20:00,348-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:20:10,072-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:20:12,593-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 09:20:20,079-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:20:30,087-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 09:20:32,672-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 09:20:40,095-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:20:48,202-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:20:50,103-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:21:00,111-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:21:04,994-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:21:10,119-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:21:20,127-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 09:21:23,813-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 09:21:30,134-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:21:37,317-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:21:40,142-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:21:50,150-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:21:59,423-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:22:00,157-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:22:09,640-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:22:10,165-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:22:20,172-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:22:30,180-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 4ms
-2013-11-06 09:22:35,036-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 4ms
-No streams
-2013-11-06 09:22:40,188-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:22:41,963-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:22:50,196-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:23:00,203-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:23:10,216-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:23:10,643-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:23:14,286-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:23:20,224-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:23:30,232-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:23:40,240-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:23:46,253-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:23:46,609-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:23:50,248-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:24:00,258-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:24:10,266-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:24:18,932-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:24:20,274-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:24:21,863-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:24:30,282-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:24:40,290-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:24:50,299-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:24:51,255-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:24:57,473-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:25:00,307-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:25:10,315-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:25:20,324-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:25:23,578-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:25:30,332-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-2013-11-06 09:25:33,084-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-No streams
-2013-11-06 09:25:40,340-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:25:50,348-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:25:55,901-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:26:00,356-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:26:08,693-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:26:10,364-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:26:20,372-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:26:28,224-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:26:30,380-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:26:40,388-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:26:44,303-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:26:50,396-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:27:00,405-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:27:00,547-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:27:10,413-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:27:19,913-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:27:20,421-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:27:30,429-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:27:32,871-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:27:40,438-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:27:50,446-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:27:55,523-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:28:00,454-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:28:05,195-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:28:10,462-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:28:20,470-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:28:30,478-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:28:31,133-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-06 09:28:37,518-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-06 09:28:40,487-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:28:50,495-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:29:00,504-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:29:06,743-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:29:09,842-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 09:29:10,512-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:29:20,519-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:29:30,527-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:29:40,534-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:29:42,165-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 09:29:42,354-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 09:29:50,542-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:30:00,550-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:30:10,558-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:30:14,488-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:30:17,963-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:30:20,565-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:30:30,574-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:30:40,582-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 09:30:46,812-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 09:30:50,589-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:30:53,573-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:31:00,597-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:31:10,605-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:31:19,134-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 09:31:20,613-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:31:29,183-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:31:30,621-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:31:40,629-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:31:50,637-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:31:51,457-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 09:32:00,645-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:32:04,793-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:32:10,656-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:32:20,665-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:32:23,780-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 09:32:30,673-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 4ms
-2013-11-06 09:32:40,406-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 4ms
-No streams
-2013-11-06 09:32:40,681-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:32:50,689-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:32:56,103-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 09:33:00,698-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:33:10,706-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:33:16,013-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:33:20,714-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:33:28,426-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 09:33:30,722-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:33:40,730-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:33:50,738-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 09:33:51,624-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:34:00,749-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 09:34:00,752-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:34:10,760-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:34:20,768-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:34:27,234-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:34:30,776-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:34:33,072-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 09:34:40,785-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:34:50,793-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:35:00,801-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:35:02,844-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:35:05,395-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 09:35:10,809-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:35:20,817-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:35:30,826-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 120ms
-2013-11-06 09:35:37,837-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 120ms
-No streams
-2013-11-06 09:35:40,834-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2525ms
-2013-11-06 09:35:40,978-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2525ms
-No streams
-2013-11-06 09:35:50,845-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:36:00,853-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:36:10,041-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 09:36:10,861-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:36:14,064-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:36:20,869-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:36:30,872-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:36:40,880-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:36:42,364-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:36:49,674-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:36:50,889-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:37:00,897-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:37:10,905-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:37:14,687-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 09:37:20,913-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-06 09:37:25,284-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-06 09:37:30,921-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:37:40,929-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:37:47,010-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 09:37:50,936-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 09:38:00,896-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 09:38:00,944-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:38:10,952-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:38:19,333-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 09:38:20,960-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:38:30,967-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:38:36,505-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:38:40,975-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:38:50,983-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-2013-11-06 09:38:51,657-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-No streams
-2013-11-06 09:39:00,991-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:39:10,998-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:39:12,116-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:39:21,006-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:39:23,979-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 09:39:31,014-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:39:41,022-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:39:47,726-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:39:51,029-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:39:56,302-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 09:40:01,038-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:40:11,045-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:40:21,053-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 09:40:23,337-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:40:28,625-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 09:40:31,061-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:40:41,068-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:40:51,076-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:40:58,946-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:41:00,948-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 09:41:01,084-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:41:11,092-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:41:21,100-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:41:31,108-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:41:33,271-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:41:34,556-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:41:41,116-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:41:51,124-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:42:01,132-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:42:05,594-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 8ms
-2013-11-06 09:42:10,173-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 8ms
-No streams
-2013-11-06 09:42:11,140-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:42:21,148-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:42:31,156-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:42:37,917-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 09:42:41,164-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 4ms
-2013-11-06 09:42:45,779-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 4ms
-No streams
-2013-11-06 09:42:51,172-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:43:01,181-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:43:10,240-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 09:43:11,189-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:43:21,197-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:43:21,386-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 09:43:31,205-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:43:41,214-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:43:42,563-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:43:51,222-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:43:56,996-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:44:01,230-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:44:11,239-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:44:14,886-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 09:44:21,247-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:44:31,255-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:44:32,606-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:44:41,264-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:44:47,209-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 09:44:51,273-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:45:01,281-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:45:08,216-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:45:11,289-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:45:19,532-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 09:45:21,298-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:45:31,306-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:45:41,314-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:45:43,826-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:45:51,322-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:45:51,855-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 09:46:01,330-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:46:11,339-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:46:19,436-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 09:46:21,347-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:46:24,178-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 09:46:31,355-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:46:41,363-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:46:51,371-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:46:55,046-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:46:56,501-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 09:47:01,379-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:47:11,388-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:47:21,396-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:47:28,824-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:47:30,656-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:47:31,404-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:47:41,412-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:47:51,421-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 20ms
-2013-11-06 09:48:01,166-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 20ms
-No streams
-2013-11-06 09:48:01,429-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 09:48:06,267-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 09:48:11,437-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:48:21,445-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:48:31,453-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:48:33,470-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 09:48:41,461-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:48:41,876-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:48:51,470-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:49:01,479-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:49:05,793-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:49:11,487-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:49:17,486-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:49:21,495-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:49:31,503-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:49:38,116-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 09:49:41,511-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:49:51,519-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:49:53,096-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:50:01,527-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:50:10,439-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:50:11,536-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:50:21,543-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:50:28,706-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 09:50:31,552-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:50:41,560-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:50:42,762-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 09:50:51,568-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:51:01,576-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:51:04,316-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:51:11,584-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:51:15,085-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 09:51:21,593-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:51:31,601-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:51:39,926-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:51:41,609-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:51:47,408-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:51:51,617-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:52:01,625-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:52:11,633-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:52:15,537-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:52:19,731-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 09:52:21,641-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:52:31,649-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:52:41,657-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 4ms
-2013-11-06 09:52:51,149-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 4ms
-No streams
-2013-11-06 09:52:51,665-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:52:52,054-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 09:53:01,673-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:53:11,683-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:53:21,691-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:53:24,377-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:53:26,756-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 09:53:31,699-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:53:41,707-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:53:51,715-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:53:56,700-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 09:54:01,723-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:54:02,366-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:54:11,731-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:54:21,740-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:54:29,023-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:54:31,747-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:54:37,976-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:54:41,755-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:54:51,763-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:55:01,346-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:55:01,771-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:55:11,779-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 09:55:13,587-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 09:55:21,787-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:55:31,795-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:55:33,669-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 09:55:41,803-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:55:49,196-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:55:51,811-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:56:01,819-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:56:05,992-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:56:11,827-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:56:21,835-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 09:56:24,807-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 09:56:31,843-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:56:38,315-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:56:41,851-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:56:51,859-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:57:00,416-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 09:57:01,866-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:57:10,638-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:57:11,874-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:57:21,882-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:57:31,890-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:57:36,026-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:57:41,898-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:57:42,961-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:57:51,906-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:58:01,914-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:58:11,636-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:58:11,923-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:58:15,284-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:58:21,931-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:58:31,939-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:58:41,947-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:58:47,246-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-06 09:58:47,607-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-06 09:58:51,955-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:59:01,963-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:59:11,971-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:59:19,931-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 09:59:21,978-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:59:22,856-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 09:59:31,986-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:59:41,994-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 09:59:52,002-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 09:59:52,254-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 09:59:58,466-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:00:02,010-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:00:12,017-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:00:22,025-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 10:00:24,578-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 10:00:32,033-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 11ms
-2013-11-06 10:00:34,086-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 11ms
-No streams
-2013-11-06 10:00:42,041-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:00:52,049-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:00:56,900-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:01:02,057-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:01:09,686-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:01:12,065-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:01:22,072-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:01:29,223-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:01:32,081-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:01:42,088-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:01:45,296-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:01:52,096-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:02:01,546-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:02:02,104-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:02:12,112-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:02:20,906-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:02:22,120-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:02:32,127-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:02:33,869-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:02:42,135-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:02:52,143-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 4ms
-2013-11-06 10:02:56,519-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 4ms
-No streams
-2013-11-06 10:03:02,151-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:03:06,192-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:03:12,159-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:03:22,167-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:03:32,126-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:03:32,175-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:03:38,515-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:03:42,182-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:03:52,190-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:04:02,199-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:04:07,736-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:04:10,838-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:04:12,207-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:04:22,215-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:04:32,223-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:04:42,231-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:04:43,161-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 10:04:43,347-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 10:04:52,239-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:05:02,247-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:05:12,256-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:05:15,484-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 10:05:18,957-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 10:05:22,264-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:05:32,272-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:05:42,280-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:05:47,807-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:05:52,289-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:05:54,566-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:06:02,297-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:06:12,306-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:06:20,130-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:06:22,313-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:06:30,176-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:06:32,321-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:06:42,329-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:06:52,337-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 10:06:52,454-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 10:07:02,345-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:07:05,786-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:07:12,356-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:07:22,366-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 10:07:24,777-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 10:07:32,376-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:07:41,396-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:07:42,386-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:07:52,395-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:07:57,099-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:08:02,406-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:08:12,415-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:08:17,006-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:08:22,423-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:08:29,422-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:08:32,435-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:08:42,447-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:08:52,461-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 10:08:52,617-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:09:01,745-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:09:02,478-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:09:12,491-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:09:22,500-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 10:09:28,227-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 10:09:32,510-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 10:09:34,069-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 10:09:42,520-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:09:52,530-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:10:02,539-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:10:03,836-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:10:06,391-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:10:12,549-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:10:22,558-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:10:32,567-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:10:38,714-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 10:10:39,447-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 10:10:42,577-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:10:52,588-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:11:02,598-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:11:11,038-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:11:12,608-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 10:11:15,057-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 10:11:22,618-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:11:32,628-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:11:42,639-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 10:11:43,361-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:11:50,667-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:11:52,649-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:12:02,657-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:12:12,666-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-2013-11-06 10:12:15,684-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-No streams
-2013-11-06 10:12:22,674-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:12:26,276-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:12:32,684-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:12:42,693-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:12:48,006-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:12:52,701-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 95ms
-2013-11-06 10:13:01,980-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 95ms
-No streams
-2013-11-06 10:13:02,709-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:13:12,717-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:13:20,329-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:13:22,725-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:13:32,733-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:13:37,496-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:13:42,741-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:13:52,652-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:13:52,749-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:14:02,757-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:14:12,765-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:14:13,106-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:14:22,773-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:14:24,975-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:14:32,781-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:14:42,789-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:14:48,716-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:14:52,801-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:14:57,298-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:15:02,809-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:15:12,816-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:15:22,824-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 10:15:24,327-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 10:15:29,622-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 10:15:32,832-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:15:42,840-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:15:52,848-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:15:59,936-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:16:01,944-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:16:02,855-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:16:12,863-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:16:22,871-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:16:32,879-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:16:34,267-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:16:35,546-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:16:42,887-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:16:52,895-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:17:02,903-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:17:06,590-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 10:17:11,157-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 10:17:12,911-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:17:22,919-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:17:32,927-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:17:38,913-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:17:42,935-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:17:46,766-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:17:52,943-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:18:02,951-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:18:11,236-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:18:12,959-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:18:22,376-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:18:22,967-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:18:32,974-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:18:42,982-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:18:43,559-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:18:52,990-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:18:57,986-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:19:02,998-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:19:13,006-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:19:15,882-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:19:23,015-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:19:33,023-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:19:33,596-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:19:43,031-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:19:48,205-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:19:53,039-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:20:03,047-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:20:09,206-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:20:13,056-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:20:20,528-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:20:23,065-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:20:33,073-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:20:43,081-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:20:44,816-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:20:52,851-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:20:53,089-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:21:03,097-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:21:13,106-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:21:20,427-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:21:23,114-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:21:25,174-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:21:33,122-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:21:43,130-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:21:53,138-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:21:56,037-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:21:57,497-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:22:03,146-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:22:13,155-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:22:23,163-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-2013-11-06 10:22:29,821-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:22:31,647-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:22:33,171-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:22:43,179-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:22:53,188-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:23:02,143-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:23:03,196-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 4ms
-2013-11-06 10:23:07,260-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 4ms
-No streams
-2013-11-06 10:23:13,204-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:23:23,212-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:23:33,220-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:23:34,466-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:23:42,867-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:23:43,228-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:23:53,236-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:24:03,245-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:24:06,789-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:24:13,253-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:24:18,478-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:24:23,261-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:24:33,269-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:24:39,112-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:24:43,278-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:24:53,286-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:24:54,088-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:25:03,294-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:25:11,435-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:25:13,303-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:25:23,311-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 10:25:29,699-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 10:25:33,319-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:25:43,327-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:25:43,758-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:25:53,335-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:26:03,343-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:26:05,308-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:26:13,351-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:26:16,081-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:26:23,359-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:26:33,367-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:26:40,918-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:26:43,375-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:26:48,404-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:26:53,382-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:27:03,390-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:27:13,398-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 10:27:16,529-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:27:20,727-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:27:23,406-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:27:33,413-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:27:43,422-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:27:52,138-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:27:53,050-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:27:53,429-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:28:03,437-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:28:13,446-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:28:23,454-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:28:25,373-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:28:27,748-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:28:33,462-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:28:43,470-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:28:53,478-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:28:57,696-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:29:03,358-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:29:03,486-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:29:13,494-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:29:23,503-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:29:30,019-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:29:33,511-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:29:38,968-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:29:43,519-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:29:53,527-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:30:02,342-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:30:03,535-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:30:13,543-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:30:14,579-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:30:23,552-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:30:33,560-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:30:34,665-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:30:43,568-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:30:50,188-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:30:53,576-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:31:03,584-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:31:06,988-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:31:13,592-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:31:23,599-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 10:31:25,799-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 10:31:33,608-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:31:39,311-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:31:43,616-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:31:53,624-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:32:01,408-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:32:03,632-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:32:11,634-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:32:13,640-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:32:23,648-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:32:33,656-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:32:37,018-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:32:43,664-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:32:43,957-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:32:53,672-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:33:03,680-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 4ms
-2013-11-06 10:33:12,631-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 4ms
-No streams
-2013-11-06 10:33:13,688-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:33:16,280-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:33:23,696-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:33:33,705-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:33:43,713-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:33:48,238-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:33:48,603-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:33:53,721-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:34:03,728-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:34:13,737-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 10:34:20,927-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 10:34:23,745-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:34:23,848-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:34:33,753-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:34:43,761-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:34:53,249-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:34:53,770-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:34:59,458-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:35:03,778-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:35:13,786-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:35:23,794-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:35:25,573-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:35:33,803-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-2013-11-06 10:35:35,069-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-No streams
-2013-11-06 10:35:43,811-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:35:53,819-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-06 10:35:57,896-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-06 10:36:03,827-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:36:10,678-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:36:13,835-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:36:23,843-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:36:30,219-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:36:33,851-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:36:43,859-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:36:46,288-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:36:53,867-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-06 10:37:02,542-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-06 10:37:03,875-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:37:13,883-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:37:21,898-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:37:23,891-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:37:33,898-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:37:34,867-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:37:43,906-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:37:53,914-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:37:57,508-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:38:03,922-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:38:07,191-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:38:13,930-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:38:23,938-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:38:33,118-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:38:33,946-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:38:39,515-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:38:43,953-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:38:53,961-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:39:03,969-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:39:08,728-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:39:11,839-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:39:13,977-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:39:23,985-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:39:33,993-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:39:44,001-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:39:44,162-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:39:44,338-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:39:54,009-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:40:04,017-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:40:14,025-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 10:40:16,486-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:40:19,948-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:40:24,033-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:40:34,041-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:40:44,049-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 10:40:48,809-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 10:40:54,057-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 10:40:55,559-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 10:41:04,065-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:41:14,074-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:41:21,132-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:41:24,082-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:41:31,168-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:41:34,089-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:41:44,097-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:41:53,455-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:41:54,106-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:42:04,114-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:42:06,778-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:42:14,122-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:42:24,130-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:42:25,779-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:42:34,141-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:42:42,388-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:42:44,149-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:42:54,157-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:42:58,102-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:43:04,166-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:43:14,174-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 3ms
-2013-11-06 10:43:18,000-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 3ms
-No streams
-2013-11-06 10:43:24,182-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:43:30,425-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:43:34,190-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:43:44,198-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-2013-11-06 10:43:53,609-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-No streams
-2013-11-06 10:43:54,206-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:44:02,748-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:44:04,214-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:44:14,222-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:44:24,230-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:44:29,218-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:44:34,238-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-06 10:44:35,072-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-06 10:44:44,246-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:44:54,254-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:45:04,262-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:45:04,828-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-06 10:45:07,394-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-06 10:45:14,270-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:45:24,278-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:45:34,286-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 10:45:39,719-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:45:40,438-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:45:44,293-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:45:54,301-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:46:04,309-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:46:12,042-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:46:14,317-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:46:16,048-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:46:24,325-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:46:34,333-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:46:44,341-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:46:44,365-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:46:51,659-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:46:54,349-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:47:04,357-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:47:14,365-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:47:16,689-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:47:24,372-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:47:27,269-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:47:34,380-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:47:44,388-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:47:49,013-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:47:54,396-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-2013-11-06 10:48:02,880-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-No streams
-2013-11-06 10:48:04,404-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:48:14,413-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:48:21,336-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:48:24,420-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:48:34,428-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:48:38,489-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:48:44,436-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:48:53,660-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:48:54,444-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:49:04,452-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:49:14,099-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:49:14,459-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:49:24,467-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 10:49:25,983-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 10:49:34,475-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:49:44,483-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:49:49,709-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:49:54,491-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-06 10:49:58,305-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-06 10:50:04,499-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:50:14,507-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:50:24,515-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 10:50:25,320-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:50:30,629-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:50:34,523-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:50:44,531-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:50:54,539-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:51:00,929-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:51:02,954-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:51:04,546-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:51:14,554-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:51:24,562-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:51:34,570-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 10:51:35,277-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:51:36,539-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:51:44,578-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:51:54,586-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:52:04,594-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 10:52:07,600-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 4ms
-2013-11-06 10:52:12,152-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 4ms
-No streams
-2013-11-06 10:52:14,602-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:52:24,610-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:52:34,620-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 10:52:39,923-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 10:52:44,627-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 10:52:47,760-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 10:52:54,634-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:53:04,642-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 10:53:12,246-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 10:53:14,649-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 4ms
-2013-11-06 10:53:23,372-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 4ms
-No streams
-2013-11-06 10:53:24,657-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:53:34,665-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:53:44,568-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:53:44,673-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:53:54,681-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:53:58,979-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:54:04,689-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:54:14,697-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:54:16,892-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:54:24,705-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:54:34,589-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:54:34,713-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:54:44,721-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:54:49,215-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:54:54,729-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:55:04,737-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:55:10,199-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:55:14,745-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 10:55:21,538-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 10:55:24,752-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:55:34,760-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:55:44,768-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 10:55:45,810-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 10:55:53,861-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 10:55:54,776-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:56:04,784-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:56:14,792-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 10:56:21,420-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 10:56:24,800-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 10:56:26,184-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 10:56:34,808-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:56:44,816-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:56:54,824-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:56:57,029-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:56:58,506-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:57:04,832-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:57:14,840-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:57:24,848-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:57:30,829-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:57:32,639-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:57:34,855-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:57:44,863-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:57:54,871-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:58:03,152-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:58:04,879-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:58:08,249-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:58:14,887-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:58:24,895-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:58:34,903-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 10:58:35,476-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:58:43,859-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:58:44,911-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:58:54,919-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:59:04,927-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:59:07,798-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:59:14,935-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:59:19,469-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 10:59:24,942-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:59:34,950-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 10:59:40,121-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 10:59:44,958-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 10:59:54,966-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 10:59:55,079-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 11:00:04,974-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 11:00:12,444-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 11:00:14,983-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:00:24,990-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 11:00:30,691-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 11:00:34,998-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 11:00:44,768-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 11:00:45,006-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:00:55,014-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:01:05,022-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:01:06,300-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 11:01:15,030-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 11:01:17,090-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 11:01:25,038-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:01:35,045-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 11:01:41,911-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 11:01:45,054-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 11:01:49,414-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 11:01:55,062-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:02:05,070-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:02:15,078-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 11:02:17,520-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 11:02:21,737-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 11:02:25,086-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:02:35,094-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:02:45,102-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 11:02:53,130-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 11:02:54,060-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 11:02:55,109-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:03:05,117-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:03:15,125-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:03:25,133-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 11:03:26,382-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 4ms
-2013-11-06 11:03:28,743-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 4ms
-No streams
-2013-11-06 11:03:35,140-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:03:45,148-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:03:55,156-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 11:03:58,705-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:04:04,350-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 11:04:05,164-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:04:15,172-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:04:25,180-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 11:04:31,028-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 11:04:35,188-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 11:04:39,960-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 11:04:45,196-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:04:55,205-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 11:05:03,351-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 11:05:05,213-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:05:15,221-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:05:15,570-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 11:05:25,229-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:05:35,237-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 11:05:35,675-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 11:05:45,245-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 11:05:51,180-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 11:05:55,253-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:06:05,261-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 11:06:07,997-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 11:06:15,269-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:06:25,276-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 11:06:26,791-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 11:06:35,284-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 11:06:40,320-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 11:06:45,292-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:06:55,300-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:07:02,400-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 11:07:05,308-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 11:07:12,643-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 11:07:15,316-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:07:25,324-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:07:35,332-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 11:07:38,010-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 11:07:44,966-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 11:07:45,340-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:07:55,348-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:08:05,356-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:08:13,620-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 11:08:15,364-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 11:08:17,289-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 11:08:25,372-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:08:35,380-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:08:45,388-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 11:08:49,230-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 11:08:49,612-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 11:08:55,396-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:09:05,404-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:09:15,412-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 11:09:21,935-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:09:24,840-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 11:09:25,420-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:09:35,428-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:09:45,436-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 11:09:54,258-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 11:09:55,444-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:10:00,450-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 11:10:05,452-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:10:15,460-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:10:25,468-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 11:10:26,582-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 11:10:35,475-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 3ms
-2013-11-06 11:10:36,062-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 3ms
-No streams
-2013-11-06 11:10:45,483-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:10:55,491-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 11:10:58,904-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 11:11:05,499-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:11:11,670-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 11:11:15,507-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:11:25,515-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 11:11:31,228-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 11:11:35,523-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:11:45,531-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 11:11:47,280-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 11:11:55,539-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 11:12:03,550-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 11:12:05,548-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:12:15,555-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:12:22,890-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 11:12:25,564-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:12:35,571-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 11:12:35,874-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 11:12:45,583-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:12:55,591-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:12:58,500-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 11:13:05,599-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:13:08,196-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 11:13:15,608-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:13:25,616-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 11:13:34,110-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 11:13:35,624-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:13:40,519-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 11:13:45,632-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:13:55,640-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:14:05,648-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:14:09,720-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 11:14:12,843-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 11:14:15,656-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:14:25,664-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:14:35,672-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 11:14:45,166-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 11:14:45,331-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 11:14:45,680-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:14:55,689-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:15:05,697-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:15:15,705-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:15:17,488-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:15:20,940-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 11:15:25,714-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:15:35,722-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:15:45,730-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 11:15:49,812-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 11:15:55,739-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 11:15:56,551-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 11:16:05,747-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:16:15,756-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 3ms
-2013-11-06 11:16:22,136-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 3ms
-No streams
-2013-11-06 11:16:25,764-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:16:32,160-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 11:16:35,772-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:16:45,780-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 11:16:54,458-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 11:16:55,789-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:17:05,797-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:17:07,770-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 11:17:15,805-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:17:25,813-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:17:26,780-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 11:17:35,821-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-06 11:17:43,380-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-06 11:17:45,829-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:17:55,837-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 11:17:59,104-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 11:18:05,846-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:18:15,854-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:18:18,991-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 11:18:25,865-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:18:31,426-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 11:18:35,873-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:18:45,881-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 3ms
-2013-11-06 11:18:54,603-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 3ms
-No streams
-2013-11-06 11:18:55,889-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-06 11:19:03,748-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-06 11:19:05,897-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:19:15,905-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:19:25,914-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:19:30,211-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 11:19:35,922-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:19:36,072-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 11:19:45,930-0600 INFO Multiplexer No streams
-Shutting down service...
-Shutting down blocks
-2013-11-06 11:19:52,259-0600 INFO BlockQueueProcessor Shutting down blocks
-Shutting down block Block crush.mcs.anl.gov (1x4611686018427388.000s)
-2013-11-06 11:19:52,260-0600 INFO Block Shutting down block Block crush.mcs.anl.gov (1x4611686018427388.000s)
-Failed to send remote log message: BLOCK_SHUTDOWN id=crush.mcs.anl.gov
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.rlog.RemoteLogger.log(RemoteLogger.java:34)
- at org.globus.cog.abstraction.coaster.service.job.manager.Block.shutdown(Block.java:283)
- at org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.shutdownBlocks(BlockQueueProcessor.java:807)
- at org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.shutdown(BlockQueueProcessor.java:796)
- at org.globus.cog.abstraction.coaster.service.job.manager.JobQueue.shutdown(JobQueue.java:119)
- at org.globus.cog.abstraction.coaster.service.CoasterService.shutdown(CoasterService.java:303)
- at org.globus.cog.abstraction.coaster.service.CoasterPersistentService$2.run(CoasterPersistentService.java:236)
-2013-11-06 11:19:52,260-0600 WARN RemoteLogger Failed to send remote log message: BLOCK_SHUTDOWN id=crush.mcs.anl.gov
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.rlog.RemoteLogger.log(RemoteLogger.java:34)
- at org.globus.cog.abstraction.coaster.service.job.manager.Block.shutdown(Block.java:283)
- at org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.shutdownBlocks(BlockQueueProcessor.java:807)
- at org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.shutdown(BlockQueueProcessor.java:796)
- at org.globus.cog.abstraction.coaster.service.job.manager.JobQueue.shutdown(JobQueue.java:119)
- at org.globus.cog.abstraction.coaster.service.CoasterService.shutdown(CoasterService.java:303)
- at org.globus.cog.abstraction.coaster.service.CoasterPersistentService$2.run(CoasterPersistentService.java:236)
-Adding short shutdown watchdog: count = 1, workers = 1, now = true
-2013-11-06 11:19:52,262-0600 INFO Block Adding short shutdown watchdog: count = 1, workers = 1, now = true
-Average utilization: 10.05%
-2013-11-06 11:19:52,262-0600 INFO Block Average utilization: 10.05%
-Failed to send remote log message: BLOCK_UTILIZATION id=crush.mcs.anl.gov, u=10.05
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.rlog.RemoteLogger.log(RemoteLogger.java:34)
- at org.globus.cog.abstraction.coaster.service.job.manager.Block.shutdown(Block.java:317)
- at org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.shutdownBlocks(BlockQueueProcessor.java:807)
- at org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.shutdown(BlockQueueProcessor.java:796)
- at org.globus.cog.abstraction.coaster.service.job.manager.JobQueue.shutdown(JobQueue.java:119)
- at org.globus.cog.abstraction.coaster.service.CoasterService.shutdown(CoasterService.java:303)
- at org.globus.cog.abstraction.coaster.service.CoasterPersistentService$2.run(CoasterPersistentService.java:236)
-2013-11-06 11:19:52,262-0600 WARN RemoteLogger Failed to send remote log message: BLOCK_UTILIZATION id=crush.mcs.anl.gov, u=10.05
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.rlog.RemoteLogger.log(RemoteLogger.java:34)
- at org.globus.cog.abstraction.coaster.service.job.manager.Block.shutdown(Block.java:317)
- at org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.shutdownBlocks(BlockQueueProcessor.java:807)
- at org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.shutdown(BlockQueueProcessor.java:796)
- at org.globus.cog.abstraction.coaster.service.job.manager.JobQueue.shutdown(JobQueue.java:119)
- at org.globus.cog.abstraction.coaster.service.CoasterService.shutdown(CoasterService.java:303)
- at org.globus.cog.abstraction.coaster.service.CoasterPersistentService$2.run(CoasterPersistentService.java:236)
-Shutting down block Block grind.mcs.anl.gov (1x4611686018427388.000s)
-2013-11-06 11:19:52,262-0600 INFO Block Shutting down block Block grind.mcs.anl.gov (1x4611686018427388.000s)
-Failed to send remote log message: BLOCK_SHUTDOWN id=grind.mcs.anl.gov
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.rlog.RemoteLogger.log(RemoteLogger.java:34)
- at org.globus.cog.abstraction.coaster.service.job.manager.Block.shutdown(Block.java:283)
- at org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.shutdownBlocks(BlockQueueProcessor.java:807)
- at org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.shutdown(BlockQueueProcessor.java:796)
- at org.globus.cog.abstraction.coaster.service.job.manager.JobQueue.shutdown(JobQueue.java:119)
- at org.globus.cog.abstraction.coaster.service.CoasterService.shutdown(CoasterService.java:303)
- at org.globus.cog.abstraction.coaster.service.CoasterPersistentService$2.run(CoasterPersistentService.java:236)
-2013-11-06 11:19:52,262-0600 WARN RemoteLogger Failed to send remote log message: BLOCK_SHUTDOWN id=grind.mcs.anl.gov
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.rlog.RemoteLogger.log(RemoteLogger.java:34)
- at org.globus.cog.abstraction.coaster.service.job.manager.Block.shutdown(Block.java:283)
- at org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.shutdownBlocks(BlockQueueProcessor.java:807)
- at org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.shutdown(BlockQueueProcessor.java:796)
- at org.globus.cog.abstraction.coaster.service.job.manager.JobQueue.shutdown(JobQueue.java:119)
- at org.globus.cog.abstraction.coaster.service.CoasterService.shutdown(CoasterService.java:303)
- at org.globus.cog.abstraction.coaster.service.CoasterPersistentService$2.run(CoasterPersistentService.java:236)
-Adding short shutdown watchdog: count = 1, workers = 1, now = true
-2013-11-06 11:19:52,263-0600 INFO Block Adding short shutdown watchdog: count = 1, workers = 1, now = true
-Average utilization: 10.12%
-2013-11-06 11:19:52,263-0600 INFO Block Average utilization: 10.12%
-Node [crush] 0 shut down successfully
-2013-11-06 11:19:52,263-0600 INFO Node Node [crush] 0 shut down successfully
-crush.mcs.anl.gov-000000: Channel shut down
-2013-11-06 11:19:52,264-0600 INFO AbstractTCPChannel crush.mcs.anl.gov-000000: Channel shut down
-MetaChannel [context: worker-1, boundTo: TCPChannel [type: server, contact: crush.mcs.anl.gov-000000]] binding to NullChannel
-2013-11-06 11:19:52,264-0600 INFO MetaChannel MetaChannel [context: worker-1, boundTo: TCPChannel [type: server, contact: crush.mcs.anl.gov-000000]] binding to NullChannel
-Using threaded sender for TCPChannel [type: server, contact: crush.mcs.anl.gov-000000]
-2013-11-06 11:19:52,264-0600 INFO AbstractStreamCoasterChannel Using threaded sender for TCPChannel [type: server, contact: crush.mcs.anl.gov-000000]
-MetaChannel [context: worker-1, boundTo: TCPChannel [type: server, contact: crush.mcs.anl.gov-000000]] binding to NullChannel
-2013-11-06 11:19:52,264-0600 INFO MetaChannel MetaChannel [context: worker-1, boundTo: TCPChannel [type: server, contact: crush.mcs.anl.gov-000000]] binding to NullChannel
-crush.mcs.anl.gov-000000: Channel terminated
-2013-11-06 11:19:52,264-0600 INFO AbstractTCPChannel crush.mcs.anl.gov-000000: Channel terminated
-Failed to send remote log message: WORKER_SHUTDOWN blockid=crush.mcs.anl.gov
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.rlog.RemoteLogger.log(RemoteLogger.java:34)
- at org.globus.cog.abstraction.coaster.service.job.manager.Node.replyReceived(Node.java:110)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 11:19:52,264-0600 WARN RemoteLogger Failed to send remote log message: WORKER_SHUTDOWN blockid=crush.mcs.anl.gov
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.rlog.RemoteLogger.log(RemoteLogger.java:34)
- at org.globus.cog.abstraction.coaster.service.job.manager.Node.replyReceived(Node.java:110)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-Node [grind] 0 shut down successfully
-2013-11-06 11:19:52,264-0600 INFO Node Node [grind] 0 shut down successfully
-grind.mcs.anl.gov-000000: Channel shut down
-2013-11-06 11:19:52,265-0600 INFO AbstractTCPChannel grind.mcs.anl.gov-000000: Channel shut down
-MetaChannel [context: worker-2, boundTo: TCPChannel [type: server, contact: grind.mcs.anl.gov-000000]] binding to NullChannel
-2013-11-06 11:19:52,265-0600 INFO MetaChannel MetaChannel [context: worker-2, boundTo: TCPChannel [type: server, contact: grind.mcs.anl.gov-000000]] binding to NullChannel
-MetaChannel [context: worker-2, boundTo: TCPChannel [type: server, contact: grind.mcs.anl.gov-000000]] binding to NullChannel
-2013-11-06 11:19:52,265-0600 INFO MetaChannel MetaChannel [context: worker-2, boundTo: TCPChannel [type: server, contact: grind.mcs.anl.gov-000000]] binding to NullChannel
-grind.mcs.anl.gov-000000: Channel terminated
-2013-11-06 11:19:52,265-0600 INFO AbstractTCPChannel grind.mcs.anl.gov-000000: Channel terminated
-Failed to send remote log message: WORKER_SHUTDOWN blockid=grind.mcs.anl.gov
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.rlog.RemoteLogger.log(RemoteLogger.java:34)
- at org.globus.cog.abstraction.coaster.service.job.manager.Node.replyReceived(Node.java:110)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 11:19:52,265-0600 WARN RemoteLogger Failed to send remote log message: WORKER_SHUTDOWN blockid=grind.mcs.anl.gov
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.rlog.RemoteLogger.log(RemoteLogger.java:34)
- at org.globus.cog.abstraction.coaster.service.job.manager.Node.replyReceived(Node.java:110)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-Failed to send remote log message: BLOCK_UTILIZATION id=grind.mcs.anl.gov, u=10.12
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.rlog.RemoteLogger.log(RemoteLogger.java:34)
- at org.globus.cog.abstraction.coaster.service.job.manager.Block.shutdown(Block.java:317)
- at org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.shutdownBlocks(BlockQueueProcessor.java:807)
- at org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.shutdown(BlockQueueProcessor.java:796)
- at org.globus.cog.abstraction.coaster.service.job.manager.JobQueue.shutdown(JobQueue.java:119)
- at org.globus.cog.abstraction.coaster.service.CoasterService.shutdown(CoasterService.java:303)
- at org.globus.cog.abstraction.coaster.service.CoasterPersistentService$2.run(CoasterPersistentService.java:236)
-2013-11-06 11:19:52,265-0600 WARN RemoteLogger Failed to send remote log message: BLOCK_UTILIZATION id=grind.mcs.anl.gov, u=10.12
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.rlog.RemoteLogger.log(RemoteLogger.java:34)
- at org.globus.cog.abstraction.coaster.service.job.manager.Block.shutdown(Block.java:317)
- at org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.shutdownBlocks(BlockQueueProcessor.java:807)
- at org.globus.cog.abstraction.coaster.service.job.manager.BlockQueueProcessor.shutdown(BlockQueueProcessor.java:796)
- at org.globus.cog.abstraction.coaster.service.job.manager.JobQueue.shutdown(JobQueue.java:119)
- at org.globus.cog.abstraction.coaster.service.CoasterService.shutdown(CoasterService.java:303)
- at org.globus.cog.abstraction.coaster.service.CoasterPersistentService$2.run(CoasterPersistentService.java:236)
-Shutdown sequence completed
-2013-11-06 11:19:52,266-0600 INFO CoasterService Shutdown sequence completed
-Running /homes/yadunand/swift-trunk/cog/modules/swift/dist/swift-svn/bin/coaster-service -nosec -port 53001 -localportfile /tmp/tmp.UbM76n032n -passive
-Switching log to: cps-2013-11-06_11-19-56.log
-2013-11-06 11:19:56,077-0600 WARN CoasterPersistentService Switching log to: cps-2013-11-06_11-19-56.log
-Local contacts: [http://140.221.8.81:36437]
-2013-11-06 11:19:56,107-0600 INFO Settings Local contacts: [http://140.221.8.81:36437]
-Starting... id=1106-1911560
-2013-11-06 11:19:56,120-0600 INFO BlockQueueProcessor Starting... id=1106-1911560
-Started local service: http://140.221.8.81:36437
-2013-11-06 11:19:56,121-0600 INFO CoasterService Started local service: http://140.221.8.81:36437
-Started coaster service: http://140.221.8.81:53001
-2013-11-06 11:19:56,121-0600 INFO CoasterService Started coaster service: http://140.221.8.81:53001
-Started coaster service: http://140.221.8.81:53001
-Worker connection URL: http://140.221.8.81:36437
-Running ssh yadunand at crush.mcs.anl.gov mkdir -p /homes/yadunand
-Running scp /homes/yadunand/swift-trunk/cog/modules/swift/dist/swift-svn/bin/worker.pl yadunand at crush.mcs.anl.gov:/homes/yadunand
-Running ssh yadunand at crush.mcs.anl.gov WORKER_LOGGING_LEVEL=DEBUG /homes/yadunand/worker.pl http://140.221.8.81:36437 crush.mcs.anl.gov /homes/yadunand/logs
-Running ssh yadunand at grind.mcs.anl.gov mkdir -p /homes/yadunand
-Running scp /homes/yadunand/swift-trunk/cog/modules/swift/dist/swift-svn/bin/worker.pl yadunand at grind.mcs.anl.gov:/homes/yadunand
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 11:19:58,142-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-Channel started: TCPChannel [type: server, contact: 140.221.8.62:40901]
-2013-11-06 11:19:58,146-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 140.221.8.62:40901]
-Received registration: blockid = crush.mcs.anl.gov, url = crush
-2013-11-06 11:19:58,151-0600 INFO LocalTCPService Received registration: blockid = crush.mcs.anl.gov, url = crush
-MetaChannel [context: worker-1, boundTo: null] binding to TCPChannel [type: server, contact: 140.221.8.62:40901]
-2013-11-06 11:19:58,163-0600 INFO MetaChannel MetaChannel [context: worker-1, boundTo: null] binding to TCPChannel [type: server, contact: 140.221.8.62:40901]
-2013-11-06 11:19:58,166-0600 DEBUG Cpu worker started: block=crush.mcs.anl.gov host=crush id=0
-2013-11-06 11:19:58,166-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-Started CPU 0:1383758398s
-2013-11-06 11:19:58,168-0600 INFO Block Started CPU 0:1383758398s
-Started worker crush.mcs.anl.gov:000000
-2013-11-06 11:19:58,168-0600 INFO Block Started worker crush.mcs.anl.gov:000000
-2013-11-06 11:19:58,168-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-Using NIO sender for TCPChannel [type: server, contact: crush.mcs.anl.gov-000000]
-2013-11-06 11:19:58,168-0600 INFO AbstractStreamCoasterChannel Using NIO sender for TCPChannel [type: server, contact: crush.mcs.anl.gov-000000]
-2013-11-06 11:19:58,169-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-Handler(tag: 32, REGISTER) unregistering (send)
-2013-11-06 11:19:58,172-0600 INFO RequestHandler Handler(tag: 32, REGISTER) unregistering (send)
-Running ssh yadunand at grind.mcs.anl.gov WORKER_LOGGING_LEVEL=DEBUG /homes/yadunand/worker.pl http://140.221.8.81:36437 grind.mcs.anl.gov /homes/yadunand/logs
-Adjusting buffer size to 8650752 for 2 sockets
-2013-11-06 11:19:58,850-0600 INFO TCPBufferManager Adjusting buffer size to 8650752 for 2 sockets
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 11:19:58,850-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-Channel started: TCPChannel [type: server, contact: 140.221.8.74:51796]
-2013-11-06 11:19:58,851-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 140.221.8.74:51796]
-Received registration: blockid = grind.mcs.anl.gov, url = grind
-2013-11-06 11:19:58,851-0600 INFO LocalTCPService Received registration: blockid = grind.mcs.anl.gov, url = grind
-MetaChannel [context: worker-2, boundTo: null] binding to TCPChannel [type: server, contact: 140.221.8.74:51796]
-2013-11-06 11:19:58,851-0600 INFO MetaChannel MetaChannel [context: worker-2, boundTo: null] binding to TCPChannel [type: server, contact: 140.221.8.74:51796]
-2013-11-06 11:19:58,851-0600 DEBUG Cpu worker started: block=grind.mcs.anl.gov host=grind id=0
-2013-11-06 11:19:58,851-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Started CPU 0:1383758398s
-2013-11-06 11:19:58,852-0600 INFO Block Started CPU 0:1383758398s
-2013-11-06 11:19:58,852-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Started worker grind.mcs.anl.gov:000000
-2013-11-06 11:19:58,852-0600 INFO Block Started worker grind.mcs.anl.gov:000000
-2013-11-06 11:19:58,852-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-Handler(tag: 7333, REGISTER) unregistering (send)
-2013-11-06 11:19:58,852-0600 INFO RequestHandler Handler(tag: 7333, REGISTER) unregistering (send)
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 11:20:31,743-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 11:20:32,288-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:21:08,762-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:21:09,057-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:21:45,235-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:21:46,375-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 11:22:21,710-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:22:23,693-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:22:58,183-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:23:01,011-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:23:34,657-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:23:38,329-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:24:11,131-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:24:15,647-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 11:24:47,606-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:24:52,965-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 11:25:24,080-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:25:30,283-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:26:00,553-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:26:07,601-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:26:37,027-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:26:44,919-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 11:27:13,502-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:27:22,237-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:27:49,975-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:27:59,555-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:28:26,449-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:28:36,873-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:29:02,923-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:29:14,191-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:29:39,397-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:29:51,509-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 11:30:15,872-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:30:28,827-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 11:30:52,346-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:31:06,145-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:31:28,820-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:31:43,463-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:32:05,294-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:32:20,781-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:32:41,768-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:32:58,099-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:33:18,242-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:33:35,417-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 11:33:54,717-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 11:34:12,735-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:34:31,190-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 11:34:34,308-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:36979] for addition
-2013-11-06 11:34:34,311-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:36979] for addition
-Multiplexer 0 started
-2013-11-06 11:34:34,311-0600 INFO Multiplexer Multiplexer 0 started
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:36979]
-2013-11-06 11:34:34,311-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:36979]
-Multiplexer 1 started
-2013-11-06 11:34:34,311-0600 INFO Multiplexer Multiplexer 1 started
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--8000
-2013-11-06 11:34:34,333-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--8000
-MetaChannel [context: cps-36979, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:36979]
-2013-11-06 11:34:34,333-0600 INFO MetaChannel MetaChannel [context: cps-36979, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:36979]
-Using threaded sender for TCPChannel [type: server, contact: 127.0.0.1:36979]
-2013-11-06 11:34:34,334-0600 INFO AbstractStreamCoasterChannel Using threaded sender for TCPChannel [type: server, contact: 127.0.0.1:36979]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 11:34:34,334-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 11:34:34,376-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-org.globus.cog.coaster.ProtocolException: Invalid line:
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler$Helper.scan(SubmitJobHandler.java:312)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler$Helper.read(SubmitJobHandler.java:270)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.read(SubmitJobHandler.java:97)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.requestComplete(SubmitJobHandler.java:61)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:173)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Handler(tag: 38908, SUBMITJOB) sending error: Could not deserialize job description
-org.globus.cog.coaster.ProtocolException: Could not deserialize job description
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.requestComplete(SubmitJobHandler.java:71)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:173)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Caused by: org.globus.cog.coaster.ProtocolException: Invalid line:
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler$Helper.scan(SubmitJobHandler.java:312)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler$Helper.read(SubmitJobHandler.java:270)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.read(SubmitJobHandler.java:97)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.requestComplete(SubmitJobHandler.java:61)
- ... 4 more
-2013-11-06 11:34:34,466-0600 INFO RequestReply Handler(tag: 38908, SUBMITJOB) sending error: Could not deserialize job description
-org.globus.cog.coaster.ProtocolException: Could not deserialize job description
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.requestComplete(SubmitJobHandler.java:71)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:173)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Caused by: org.globus.cog.coaster.ProtocolException: Invalid line:
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler$Helper.scan(SubmitJobHandler.java:312)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler$Helper.read(SubmitJobHandler.java:270)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.read(SubmitJobHandler.java:97)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.requestComplete(SubmitJobHandler.java:61)
- ... 4 more
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 11:34:34,477-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-Avg stream buf: 0
-2013-11-06 11:34:44,328-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:34:50,053-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 11:34:54,339-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 11:35:04,350-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:35:07,664-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 11:35:14,361-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 11:35:24,372-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:35:27,371-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 11:35:34,380-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:35:44,138-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 11:35:44,388-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 11:35:54,398-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 11:36:04,407-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:36:04,689-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 11:36:14,416-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:36:20,612-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 11:36:24,425-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 11:36:34,434-0600 INFO Multiplexer Avg stream buf: 0
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-113434.459, now=131106-113635.309, channel=TCPChannel [type: server, contact: 127.0.0.1:36979]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 11:36:35,309-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-113434.459, now=131106-113635.309, channel=TCPChannel [type: server, contact: 127.0.0.1:36979]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 11:36:35,309-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:36979] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-113434.459, now=131106-113635.309, channel=TCPChannel [type: server, contact: 127.0.0.1:36979]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 11:36:35,309-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:36979] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-113434.459, now=131106-113635.309, channel=TCPChannel [type: server, contact: 127.0.0.1:36979]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-null: Channel shut down
-2013-11-06 11:36:35,310-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-36979, boundTo: TCPChannel [type: server, contact: 127.0.0.1:36979]] binding to NullChannel
-2013-11-06 11:36:35,313-0600 INFO MetaChannel MetaChannel [context: cps-36979, boundTo: TCPChannel [type: server, contact: 127.0.0.1:36979]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:36979]
-2013-11-06 11:36:35,313-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:36979]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-113434.459, now=131106-113635.309, channel=TCPChannel [type: server, contact: 127.0.0.1:36979]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 11:36:35,313-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-113434.459, now=131106-113635.309, channel=TCPChannel [type: server, contact: 127.0.0.1:36979]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel exception handled
-2013-11-06 11:36:35,313-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 11:36:35,317-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 11:36:35,317-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 11:36:35,317-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:36979] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 11:36:35,317-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:36979] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 11:36:35,318-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:36979]
-2013-11-06 11:36:35,318-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:36979]
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:36:42,007-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 11:36:44,443-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:36:54,451-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:36:57,086-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 11:37:04,459-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:37:14,467-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 11:37:19,326-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 11:37:24,475-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:37:33,561-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 11:37:34,483-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:37:44,491-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:37:54,498-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:37:56,643-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 11:38:04,506-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:38:10,035-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 11:38:14,513-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:38:24,521-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 11:38:33,961-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 11:38:34,529-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:38:44,536-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:38:46,510-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 11:38:54,544-0600 INFO Multiplexer No streams
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 11:39:03,519-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:36980] for addition
-2013-11-06 11:39:03,519-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:36980] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:36980]
-2013-11-06 11:39:03,519-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:36980]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fff
-2013-11-06 11:39:03,550-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fff
-MetaChannel [context: cps-36980, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:36980]
-2013-11-06 11:39:03,550-0600 INFO MetaChannel MetaChannel [context: cps-36980, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:36980]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 11:39:03,550-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 11:39:03,591-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 11:39:03,591-0600 INFO Sender Sender TCPChannel queue size: 0
-org.globus.cog.coaster.ProtocolException: Invalid line:
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler$Helper.scan(SubmitJobHandler.java:312)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler$Helper.read(SubmitJobHandler.java:270)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.read(SubmitJobHandler.java:97)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.requestComplete(SubmitJobHandler.java:61)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:173)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Handler(tag: 38908, SUBMITJOB) sending error: Could not deserialize job description
-org.globus.cog.coaster.ProtocolException: Could not deserialize job description
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.requestComplete(SubmitJobHandler.java:71)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:173)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Caused by: org.globus.cog.coaster.ProtocolException: Invalid line:
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler$Helper.scan(SubmitJobHandler.java:312)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler$Helper.read(SubmitJobHandler.java:270)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.read(SubmitJobHandler.java:97)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.requestComplete(SubmitJobHandler.java:61)
- ... 4 more
-2013-11-06 11:39:03,672-0600 INFO RequestReply Handler(tag: 38908, SUBMITJOB) sending error: Could not deserialize job description
-org.globus.cog.coaster.ProtocolException: Could not deserialize job description
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.requestComplete(SubmitJobHandler.java:71)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:173)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Caused by: org.globus.cog.coaster.ProtocolException: Invalid line:
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler$Helper.scan(SubmitJobHandler.java:312)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler$Helper.read(SubmitJobHandler.java:270)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.read(SubmitJobHandler.java:97)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.requestComplete(SubmitJobHandler.java:61)
- ... 4 more
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 11:39:03,673-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-Avg stream buf: 0
-2013-11-06 11:39:04,555-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 11:39:11,280-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 11:39:14,564-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:39:22,984-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 11:39:24,573-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 11:39:34,583-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 11:39:44,592-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:39:48,597-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 11:39:54,600-0600 INFO Multiplexer Avg stream buf: 0
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 11:39:55,502-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:36981] for addition
-2013-11-06 11:39:55,503-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:36981] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:36981]
-2013-11-06 11:39:55,503-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:36981]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7ffe
-2013-11-06 11:39:55,524-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7ffe
-MetaChannel [context: cps-36981, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:36981]
-2013-11-06 11:39:55,524-0600 INFO MetaChannel MetaChannel [context: cps-36981, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:36981]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 11:39:55,525-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 11:39:55,565-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 11:39:55,565-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 11:39:55,653-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 11:39:55,682-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 1.0) submitting urn:job-0-1383759274468-1383759274469: /bin/echo
-2013-11-06 11:39:55,683-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 1.0) submitting urn:job-0-1383759274468-1383759274469: /bin/echo
-2013-11-06 11:39:55,687-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 11:39:55,687-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-LOG: /homes/yadunand/logs/worker-grind.mcs.anl.gov.log
-2013-11-06 11:39:55,696-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7334, JOBSTATUS) unregistering (send)
-2013-11-06 11:39:55,697-0600 INFO RequestHandler Handler(tag: 7334, JOBSTATUS) unregistering (send)
-2013-11-06 11:39:55,703-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 11:39:55,703-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 11:39:55,703-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Handler(tag: 7335, JOBSTATUS) unregistering (send)
-2013-11-06 11:39:55,704-0600 INFO RequestHandler Handler(tag: 7335, JOBSTATUS) unregistering (send)
-2013-11-06 11:39:55,704-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-2013-11-06 11:39:55,704-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:39:59,458-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 11:40:04,601-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 11:40:14,612-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 11:40:24,623-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:40:25,915-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 11:40:34,633-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 3ms
-2013-11-06 11:40:35,934-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 3ms
-Avg stream buf: 0
-2013-11-06 11:40:44,643-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 11:40:54,653-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 11:41:03,233-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-113903.671, now=131106-114104.519, channel=TCPChannel [type: server, contact: 127.0.0.1:36980]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 11:41:04,519-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-113903.671, now=131106-114104.519, channel=TCPChannel [type: server, contact: 127.0.0.1:36980]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 11:41:04,520-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:36980] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-113903.671, now=131106-114104.519, channel=TCPChannel [type: server, contact: 127.0.0.1:36980]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 11:41:04,520-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:36980] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-113903.671, now=131106-114104.519, channel=TCPChannel [type: server, contact: 127.0.0.1:36980]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-null: Channel shut down
-2013-11-06 11:41:04,520-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-36980, boundTo: TCPChannel [type: server, contact: 127.0.0.1:36980]] binding to NullChannel
-2013-11-06 11:41:04,520-0600 INFO MetaChannel MetaChannel [context: cps-36980, boundTo: TCPChannel [type: server, contact: 127.0.0.1:36980]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:36980]
-2013-11-06 11:41:04,520-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:36980]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-113903.671, now=131106-114104.519, channel=TCPChannel [type: server, contact: 127.0.0.1:36980]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 11:41:04,520-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-113903.671, now=131106-114104.519, channel=TCPChannel [type: server, contact: 127.0.0.1:36980]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel exception handled
-2013-11-06 11:41:04,521-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 11:41:04,522-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 11:41:04,522-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 11:41:04,523-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:36980] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 11:41:04,523-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:36980] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 11:41:04,523-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:36980]
-2013-11-06 11:41:04,523-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:36980]
-Avg stream buf: 0
-2013-11-06 11:41:04,664-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:41:12,406-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 11:41:14,672-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 11:41:24,681-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 11:41:34,690-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 11:41:40,551-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 11:41:44,699-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:41:48,880-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 11:41:54,708-0600 INFO Multiplexer Avg stream buf: 0
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-113955.645, now=131106-114156.502, channel=TCPChannel [type: server, contact: 127.0.0.1:36981]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 11:41:56,502-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-113955.645, now=131106-114156.502, channel=TCPChannel [type: server, contact: 127.0.0.1:36981]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, RLOG)
-2013-11-06 11:41:56,502-0600 INFO ChannelContext => Command(tag: 1, RLOG)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 11:41:56,502-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-113955.645, now=131106-114156.502, channel=TCPChannel [type: server, contact: 127.0.0.1:36981]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 11:41:56,503-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-113955.645, now=131106-114156.502, channel=TCPChannel [type: server, contact: 127.0.0.1:36981]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 11:41:56,503-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-113955.645, now=131106-114156.502, channel=TCPChannel [type: server, contact: 127.0.0.1:36981]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 11:41:56,503-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-113955.645, now=131106-114156.502, channel=TCPChannel [type: server, contact: 127.0.0.1:36981]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 11:41:56,503-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-113955.645, now=131106-114156.502, channel=TCPChannel [type: server, contact: 127.0.0.1:36981]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 11:41:56,503-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-113955.645, now=131106-114156.502, channel=TCPChannel [type: server, contact: 127.0.0.1:36981]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 5, JOBSTATUS)
-2013-11-06 11:41:56,504-0600 INFO ChannelContext => Command(tag: 5, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-113955.645, now=131106-114156.502, channel=TCPChannel [type: server, contact: 127.0.0.1:36981]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 11:41:56,504-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-113955.645, now=131106-114156.502, channel=TCPChannel [type: server, contact: 127.0.0.1:36981]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 11:41:56,504-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:36981] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-113955.645, now=131106-114156.502, channel=TCPChannel [type: server, contact: 127.0.0.1:36981]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 11:41:56,504-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:36981] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-113955.645, now=131106-114156.502, channel=TCPChannel [type: server, contact: 127.0.0.1:36981]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-null: Channel shut down
-2013-11-06 11:41:56,504-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-36981, boundTo: TCPChannel [type: server, contact: 127.0.0.1:36981]] binding to NullChannel
-2013-11-06 11:41:56,505-0600 INFO MetaChannel MetaChannel [context: cps-36981, boundTo: TCPChannel [type: server, contact: 127.0.0.1:36981]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:36981]
-2013-11-06 11:41:56,505-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:36981]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-113955.645, now=131106-114156.502, channel=TCPChannel [type: server, contact: 127.0.0.1:36981]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 11:41:56,505-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-113955.645, now=131106-114156.502, channel=TCPChannel [type: server, contact: 127.0.0.1:36981]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, RLOG)
-2013-11-06 11:41:56,505-0600 INFO ChannelContext => Command(tag: 1, RLOG)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 11:41:56,505-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-113955.645, now=131106-114156.502, channel=TCPChannel [type: server, contact: 127.0.0.1:36981]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 11:41:56,505-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-113955.645, now=131106-114156.502, channel=TCPChannel [type: server, contact: 127.0.0.1:36981]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 11:41:56,505-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-113955.645, now=131106-114156.502, channel=TCPChannel [type: server, contact: 127.0.0.1:36981]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 11:41:56,505-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-113955.645, now=131106-114156.502, channel=TCPChannel [type: server, contact: 127.0.0.1:36981]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 11:41:56,506-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-113955.645, now=131106-114156.502, channel=TCPChannel [type: server, contact: 127.0.0.1:36981]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 11:41:56,506-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-113955.645, now=131106-114156.502, channel=TCPChannel [type: server, contact: 127.0.0.1:36981]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 5, JOBSTATUS)
-2013-11-06 11:41:56,506-0600 INFO ChannelContext => Command(tag: 5, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-113955.645, now=131106-114156.502, channel=TCPChannel [type: server, contact: 127.0.0.1:36981]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 11:41:56,506-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-113955.645, now=131106-114156.502, channel=TCPChannel [type: server, contact: 127.0.0.1:36981]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel exception handled
-2013-11-06 11:41:56,506-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 11:41:56,513-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 11:41:56,513-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 11:41:56,514-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:36981] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 11:41:56,514-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:36981] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 11:41:56,514-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:36981]
-2013-11-06 11:41:56,514-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:36981]
-No streams
-2013-11-06 11:42:04,717-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:42:14,726-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:42:17,869-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 11:42:24,734-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:42:25,354-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 11:42:34,742-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:42:44,751-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:42:54,759-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:42:55,187-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:43:01,828-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 11:43:04,767-0600 INFO Multiplexer No streams
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 11:43:12,592-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:36982] for addition
-2013-11-06 11:43:12,592-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:36982] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:36982]
-2013-11-06 11:43:12,592-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:36982]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7ffd
-2013-11-06 11:43:12,630-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7ffd
-MetaChannel [context: cps-36982, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:36982]
-2013-11-06 11:43:12,630-0600 INFO MetaChannel MetaChannel [context: cps-36982, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:36982]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 11:43:12,631-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 11:43:12,651-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 11:43:12,651-0600 INFO Sender Sender TCPChannel queue size: 0
-org.globus.cog.coaster.ProtocolException: Invalid line:
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler$Helper.scan(SubmitJobHandler.java:312)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler$Helper.read(SubmitJobHandler.java:270)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.read(SubmitJobHandler.java:97)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.requestComplete(SubmitJobHandler.java:61)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:173)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Handler(tag: 38908, SUBMITJOB) sending error: Could not deserialize job description
-org.globus.cog.coaster.ProtocolException: Could not deserialize job description
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.requestComplete(SubmitJobHandler.java:71)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:173)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Caused by: org.globus.cog.coaster.ProtocolException: Invalid line:
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler$Helper.scan(SubmitJobHandler.java:312)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler$Helper.read(SubmitJobHandler.java:270)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.read(SubmitJobHandler.java:97)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.requestComplete(SubmitJobHandler.java:61)
- ... 4 more
-2013-11-06 11:43:12,732-0600 INFO RequestReply Handler(tag: 38908, SUBMITJOB) sending error: Could not deserialize job description
-org.globus.cog.coaster.ProtocolException: Could not deserialize job description
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.requestComplete(SubmitJobHandler.java:71)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:173)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Caused by: org.globus.cog.coaster.ProtocolException: Invalid line:
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler$Helper.scan(SubmitJobHandler.java:312)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler$Helper.read(SubmitJobHandler.java:270)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.read(SubmitJobHandler.java:97)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.requestComplete(SubmitJobHandler.java:61)
- ... 4 more
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 11:43:12,733-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-Avg stream buf: 0
-2013-11-06 11:43:14,779-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 11:43:24,788-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 11:43:32,505-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-Avg stream buf: 0
-2013-11-06 11:43:34,796-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:43:38,302-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 11:43:44,804-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 11:43:54,813-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 11:44:04,821-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:44:09,824-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:44:14,776-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 11:44:14,829-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 11:44:24,836-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 11:44:34,844-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 11:44:44,852-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:44:47,142-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:44:51,250-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 11:44:54,860-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 11:45:04,868-0600 INFO Multiplexer Avg stream buf: 0
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-114312.731, now=131106-114513.592, channel=TCPChannel [type: server, contact: 127.0.0.1:36982]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 11:45:13,592-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-114312.731, now=131106-114513.592, channel=TCPChannel [type: server, contact: 127.0.0.1:36982]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 11:45:13,592-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:36982] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-114312.731, now=131106-114513.592, channel=TCPChannel [type: server, contact: 127.0.0.1:36982]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 11:45:13,593-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:36982] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-114312.731, now=131106-114513.592, channel=TCPChannel [type: server, contact: 127.0.0.1:36982]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-null: Channel shut down
-2013-11-06 11:45:13,593-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-36982, boundTo: TCPChannel [type: server, contact: 127.0.0.1:36982]] binding to NullChannel
-2013-11-06 11:45:13,593-0600 INFO MetaChannel MetaChannel [context: cps-36982, boundTo: TCPChannel [type: server, contact: 127.0.0.1:36982]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:36982]
-2013-11-06 11:45:13,593-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:36982]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-114312.731, now=131106-114513.592, channel=TCPChannel [type: server, contact: 127.0.0.1:36982]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 11:45:13,593-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-114312.731, now=131106-114513.592, channel=TCPChannel [type: server, contact: 127.0.0.1:36982]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel exception handled
-2013-11-06 11:45:13,593-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 11:45:13,612-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 11:45:13,612-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 11:45:13,612-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:36982] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 11:45:13,612-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:36982] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 11:45:13,612-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:36982]
-2013-11-06 11:45:13,613-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:36982]
-No streams
-2013-11-06 11:45:14,876-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 11:45:24,460-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 11:45:24,883-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:45:27,724-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 11:45:34,891-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:45:44,898-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:45:54,905-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:46:01,778-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:46:04,198-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 11:46:04,913-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:46:14,920-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:46:24,927-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:46:34,935-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 11:46:39,096-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:46:40,672-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 11:46:44,942-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:46:54,950-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:47:04,957-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:47:14,964-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:47:16,414-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 11:47:17,147-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 11:47:24,972-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:47:34,979-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:47:44,987-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:47:53,620-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:47:53,732-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 11:47:54,996-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:48:05,004-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:48:15,012-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:48:25,020-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:48:30,094-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:48:31,050-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 11:48:35,028-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:48:45,036-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:48:55,044-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:49:05,052-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:49:06,568-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:49:08,368-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 11:49:15,060-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:49:25,067-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:49:35,075-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 11:49:43,043-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 11:49:45,083-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:49:45,686-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 11:49:55,091-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:50:05,099-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:50:15,107-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 11:50:19,517-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:50:23,004-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 11:50:25,115-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:50:35,123-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:50:45,130-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:50:55,138-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 11:50:55,991-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 11:50:57,680-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:36983] for addition
-2013-11-06 11:50:57,680-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:36983] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:36983]
-2013-11-06 11:50:57,680-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:36983]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7ffc
-2013-11-06 11:50:57,706-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7ffc
-MetaChannel [context: cps-36983, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:36983]
-2013-11-06 11:50:57,707-0600 INFO MetaChannel MetaChannel [context: cps-36983, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:36983]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 11:50:57,707-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 11:50:57,747-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 11:50:57,747-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-org.globus.cog.coaster.ProtocolException: Invalid line:
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler$Helper.scan(SubmitJobHandler.java:312)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler$Helper.read(SubmitJobHandler.java:270)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.read(SubmitJobHandler.java:97)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.requestComplete(SubmitJobHandler.java:61)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:173)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Handler(tag: 38908, SUBMITJOB) sending error: Could not deserialize job description
-org.globus.cog.coaster.ProtocolException: Could not deserialize job description
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.requestComplete(SubmitJobHandler.java:71)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:173)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Caused by: org.globus.cog.coaster.ProtocolException: Invalid line:
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler$Helper.scan(SubmitJobHandler.java:312)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler$Helper.read(SubmitJobHandler.java:270)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.read(SubmitJobHandler.java:97)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.requestComplete(SubmitJobHandler.java:61)
- ... 4 more
-2013-11-06 11:50:57,828-0600 INFO RequestReply Handler(tag: 38908, SUBMITJOB) sending error: Could not deserialize job description
-org.globus.cog.coaster.ProtocolException: Could not deserialize job description
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.requestComplete(SubmitJobHandler.java:71)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:173)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Caused by: org.globus.cog.coaster.ProtocolException: Invalid line:
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler$Helper.scan(SubmitJobHandler.java:312)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler$Helper.read(SubmitJobHandler.java:270)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.read(SubmitJobHandler.java:97)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.requestComplete(SubmitJobHandler.java:61)
- ... 4 more
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 11:50:57,829-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:51:00,322-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 11:51:05,150-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 11:51:15,158-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 11:51:25,166-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:51:32,464-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 11:51:35,175-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:51:37,640-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 11:51:45,183-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 11:51:55,191-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 11:52:05,200-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 11:52:08,939-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 11:52:11,733-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:36989] for addition
-2013-11-06 11:52:11,733-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:36989] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:36989]
-2013-11-06 11:52:11,733-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:36989]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7ffb
-2013-11-06 11:52:11,760-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7ffb
-MetaChannel [context: cps-36989, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:36989]
-2013-11-06 11:52:11,760-0600 INFO MetaChannel MetaChannel [context: cps-36989, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:36989]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 11:52:11,760-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 11:52:11,801-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 11:52:11,801-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-org.globus.cog.coaster.ProtocolException: Invalid line:
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler$Helper.scan(SubmitJobHandler.java:312)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler$Helper.read(SubmitJobHandler.java:270)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.read(SubmitJobHandler.java:97)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.requestComplete(SubmitJobHandler.java:61)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:173)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Handler(tag: 38908, SUBMITJOB) sending error: Could not deserialize job description
-org.globus.cog.coaster.ProtocolException: Could not deserialize job description
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.requestComplete(SubmitJobHandler.java:71)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:173)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Caused by: org.globus.cog.coaster.ProtocolException: Invalid line:
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler$Helper.scan(SubmitJobHandler.java:312)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler$Helper.read(SubmitJobHandler.java:270)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.read(SubmitJobHandler.java:97)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.requestComplete(SubmitJobHandler.java:61)
- ... 4 more
-2013-11-06 11:52:11,882-0600 INFO RequestReply Handler(tag: 38908, SUBMITJOB) sending error: Could not deserialize job description
-org.globus.cog.coaster.ProtocolException: Could not deserialize job description
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.requestComplete(SubmitJobHandler.java:71)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:173)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Caused by: org.globus.cog.coaster.ProtocolException: Invalid line:
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler$Helper.scan(SubmitJobHandler.java:312)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler$Helper.read(SubmitJobHandler.java:270)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.read(SubmitJobHandler.java:97)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.requestComplete(SubmitJobHandler.java:61)
- ... 4 more
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 11:52:11,883-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:52:14,958-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 11:52:15,212-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 11:52:25,217-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 11:52:35,218-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 11:52:45,228-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:52:45,412-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:52:52,276-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 11:52:55,238-0600 INFO Multiplexer Avg stream buf: 0
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-115057.828, now=131106-115258.681, channel=TCPChannel [type: server, contact: 127.0.0.1:36983]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 11:52:58,681-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-115057.828, now=131106-115258.681, channel=TCPChannel [type: server, contact: 127.0.0.1:36983]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 11:52:58,681-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:36983] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-115057.828, now=131106-115258.681, channel=TCPChannel [type: server, contact: 127.0.0.1:36983]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 11:52:58,681-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:36983] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-115057.828, now=131106-115258.681, channel=TCPChannel [type: server, contact: 127.0.0.1:36983]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-null: Channel shut down
-2013-11-06 11:52:58,682-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-36983, boundTo: TCPChannel [type: server, contact: 127.0.0.1:36983]] binding to NullChannel
-2013-11-06 11:52:58,682-0600 INFO MetaChannel MetaChannel [context: cps-36983, boundTo: TCPChannel [type: server, contact: 127.0.0.1:36983]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:36983]
-2013-11-06 11:52:58,682-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:36983]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-115057.828, now=131106-115258.681, channel=TCPChannel [type: server, contact: 127.0.0.1:36983]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 11:52:58,682-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-115057.828, now=131106-115258.681, channel=TCPChannel [type: server, contact: 127.0.0.1:36983]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel exception handled
-2013-11-06 11:52:58,682-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 11:52:58,688-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 11:52:58,688-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 11:52:58,688-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:36983] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 11:52:58,688-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:36983] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 11:52:58,689-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:36983]
-2013-11-06 11:52:58,689-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:36983]
-Avg stream buf: 0
-2013-11-06 11:53:05,248-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 11:53:15,256-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:53:21,886-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 11:53:25,265-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:53:29,594-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 11:53:35,273-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 11:53:45,281-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 11:53:55,289-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:53:58,360-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 11:54:05,297-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:54:06,912-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-115211.881, now=131106-115412.734, channel=TCPChannel [type: server, contact: 127.0.0.1:36989]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 11:54:12,734-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-115211.881, now=131106-115412.734, channel=TCPChannel [type: server, contact: 127.0.0.1:36989]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 11:54:12,734-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:36989] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-115211.881, now=131106-115412.734, channel=TCPChannel [type: server, contact: 127.0.0.1:36989]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 11:54:12,734-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:36989] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-115211.881, now=131106-115412.734, channel=TCPChannel [type: server, contact: 127.0.0.1:36989]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-null: Channel shut down
-2013-11-06 11:54:12,735-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-36989, boundTo: TCPChannel [type: server, contact: 127.0.0.1:36989]] binding to NullChannel
-2013-11-06 11:54:12,735-0600 INFO MetaChannel MetaChannel [context: cps-36989, boundTo: TCPChannel [type: server, contact: 127.0.0.1:36989]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:36989]
-2013-11-06 11:54:12,735-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:36989]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-115211.881, now=131106-115412.734, channel=TCPChannel [type: server, contact: 127.0.0.1:36989]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 11:54:12,735-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-115211.881, now=131106-115412.734, channel=TCPChannel [type: server, contact: 127.0.0.1:36989]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel exception handled
-2013-11-06 11:54:12,735-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 11:54:12,738-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 11:54:12,738-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 11:54:12,738-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:36989] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 11:54:12,739-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:36989] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 11:54:12,739-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:36989]
-2013-11-06 11:54:12,739-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:36989]
-No streams
-2013-11-06 11:54:15,306-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:54:25,313-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:54:34,834-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 11:54:35,321-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 11:54:44,231-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 11:54:45,328-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:54:55,336-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:55:05,343-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:55:11,308-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 11:55:15,351-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:55:21,548-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 11:55:25,358-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 11:55:35,366-0600 INFO Multiplexer No streams
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 11:55:37,527-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:36990] for addition
-2013-11-06 11:55:37,527-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:36990] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:36990]
-2013-11-06 11:55:37,527-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:36990]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7ffa
-2013-11-06 11:55:37,552-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7ffa
-MetaChannel [context: cps-36990, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:36990]
-2013-11-06 11:55:37,552-0600 INFO MetaChannel MetaChannel [context: cps-36990, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:36990]
-Handler(tag: 17767, CHANNELCONFIG) unregistering (send)
-2013-11-06 11:55:37,553-0600 INFO RequestHandler Handler(tag: 17767, CHANNELCONFIG) unregistering (send)
-Avg stream buf: 0
-2013-11-06 11:55:45,375-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 11:55:47,783-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-Avg stream buf: 0
-2013-11-06 11:55:55,383-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 11:55:58,866-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 11:56:03,523-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:36991] for addition
-2013-11-06 11:56:03,523-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:36991] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:36991]
-2013-11-06 11:56:03,524-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:36991]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7ff9
-2013-11-06 11:56:03,546-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7ff9
-MetaChannel [context: cps-36991, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:36991]
-2013-11-06 11:56:03,546-0600 INFO MetaChannel MetaChannel [context: cps-36991, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:36991]
-Handler(tag: 17767, CHANNELCONFIG) unregistering (send)
-2013-11-06 11:56:03,546-0600 INFO RequestHandler Handler(tag: 17767, CHANNELCONFIG) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 11:56:03,547-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 17768, CONFIGSERVICE) unregistering (send)
-2013-11-06 11:56:03,587-0600 INFO RequestHandler Handler(tag: 17768, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 11:56:03,587-0600 INFO Sender Sender TCPChannel queue size: 0
-Avg stream buf: 0
-2013-11-06 11:56:05,392-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 11:56:15,401-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 3ms
-2013-11-06 11:56:24,258-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 3ms
-Avg stream buf: 0
-2013-11-06 11:56:25,410-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 11:56:35,419-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:56:36,185-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 11:56:45,428-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 11:56:55,436-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:57:00,730-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 11:57:05,445-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:57:13,502-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 11:57:15,453-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 11:57:25,461-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 11:57:35,470-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 11:57:37,205-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-115537.552, now=131106-115738.527, channel=TCPChannel [type: server, contact: 127.0.0.1:36990]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 11:57:38,527-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-115537.552, now=131106-115738.527, channel=TCPChannel [type: server, contact: 127.0.0.1:36990]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 11:57:38,528-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:36990] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-115537.552, now=131106-115738.527, channel=TCPChannel [type: server, contact: 127.0.0.1:36990]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 11:57:38,528-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:36990] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-115537.552, now=131106-115738.527, channel=TCPChannel [type: server, contact: 127.0.0.1:36990]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-null: Channel shut down
-2013-11-06 11:57:38,528-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-36990, boundTo: TCPChannel [type: server, contact: 127.0.0.1:36990]] binding to NullChannel
-2013-11-06 11:57:38,528-0600 INFO MetaChannel MetaChannel [context: cps-36990, boundTo: TCPChannel [type: server, contact: 127.0.0.1:36990]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:36990]
-2013-11-06 11:57:38,528-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:36990]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-115537.552, now=131106-115738.527, channel=TCPChannel [type: server, contact: 127.0.0.1:36990]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 11:57:38,529-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-115537.552, now=131106-115738.527, channel=TCPChannel [type: server, contact: 127.0.0.1:36990]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel exception handled
-2013-11-06 11:57:38,529-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 11:57:38,538-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 11:57:38,539-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 11:57:38,539-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:36990] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 11:57:38,539-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:36990] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 11:57:38,539-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:36990]
-2013-11-06 11:57:38,539-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:36990]
-Avg stream buf: 0
-2013-11-06 11:57:45,479-0600 INFO Multiplexer Avg stream buf: 0
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 11:57:45,839-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:36992] for addition
-2013-11-06 11:57:45,839-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:36992] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:36992]
-2013-11-06 11:57:45,839-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:36992]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7ff8
-2013-11-06 11:57:45,860-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7ff8
-MetaChannel [context: cps-36992, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:36992]
-2013-11-06 11:57:45,861-0600 INFO MetaChannel MetaChannel [context: cps-36992, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:36992]
-Handler(tag: 17767, CHANNELCONFIG) unregistering (send)
-2013-11-06 11:57:45,861-0600 INFO RequestHandler Handler(tag: 17767, CHANNELCONFIG) unregistering (send)
-Handler(tag: 17768, CONFIGSERVICE) unregistering (send)
-2013-11-06 11:57:45,901-0600 INFO RequestHandler Handler(tag: 17768, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 11:57:45,901-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 17769, SUBMITJOB) unregistering (send)
-2013-11-06 11:57:45,982-0600 INFO RequestHandler Handler(tag: 17769, SUBMITJOB) unregistering (send)
-2013-11-06 11:57:46,008-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 2.0) submitting urn:job-0-1383759274480-1383759274481: /bin/bash /homes/yadunand/swift-trunk/cog/modules/provider-coaster-c-client/tests/tcl/wrapper.sh
-2013-11-06 11:57:46,008-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 2.0) submitting urn:job-0-1383759274480-1383759274481: /bin/bash /homes/yadunand/swift-trunk/cog/modules/provider-coaster-c-client/tests/tcl/wrapper.sh
-2013-11-06 11:57:46,009-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 11:57:46,009-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-06 11:57:46,015-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7336, JOBSTATUS) unregistering (send)
-2013-11-06 11:57:46,016-0600 INFO RequestHandler Handler(tag: 7336, JOBSTATUS) unregistering (send)
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:57:50,820-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:57:51,326-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 11:57:51,326-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 11:57:51,326-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-06 11:57:51,326-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7337, JOBSTATUS) unregistering (send)
-2013-11-06 11:57:51,326-0600 INFO RequestHandler Handler(tag: 7337, JOBSTATUS) unregistering (send)
-2013-11-06 11:57:51,326-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-Avg stream buf: 0
-2013-11-06 11:57:55,482-0600 INFO Multiplexer Avg stream buf: 0
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-115603.587, now=131106-115804.523, channel=TCPChannel [type: server, contact: 127.0.0.1:36991]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 11:58:04,523-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-115603.587, now=131106-115804.523, channel=TCPChannel [type: server, contact: 127.0.0.1:36991]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 11:58:04,523-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:36991] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-115603.587, now=131106-115804.523, channel=TCPChannel [type: server, contact: 127.0.0.1:36991]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 11:58:04,524-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:36991] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-115603.587, now=131106-115804.523, channel=TCPChannel [type: server, contact: 127.0.0.1:36991]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-null: Channel shut down
-2013-11-06 11:58:04,524-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-36991, boundTo: TCPChannel [type: server, contact: 127.0.0.1:36991]] binding to NullChannel
-2013-11-06 11:58:04,524-0600 INFO MetaChannel MetaChannel [context: cps-36991, boundTo: TCPChannel [type: server, contact: 127.0.0.1:36991]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:36991]
-2013-11-06 11:58:04,524-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:36991]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-115603.587, now=131106-115804.523, channel=TCPChannel [type: server, contact: 127.0.0.1:36991]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 11:58:04,524-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-115603.587, now=131106-115804.523, channel=TCPChannel [type: server, contact: 127.0.0.1:36991]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel exception handled
-2013-11-06 11:58:04,524-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 11:58:04,527-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 11:58:04,527-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 11:58:04,527-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:36991] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 11:58:04,527-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:36991] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 11:58:04,527-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:36991]
-2013-11-06 11:58:04,527-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:36991]
-Avg stream buf: 0
-2013-11-06 11:58:05,490-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 11:58:13,679-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-Avg stream buf: 0
-2013-11-06 11:58:15,498-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 11:58:25,507-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:58:28,138-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 11:58:35,515-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 11:58:45,524-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:58:50,153-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 11:58:55,533-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:59:05,456-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 11:59:05,541-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 11:59:15,550-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 11:59:25,559-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:59:26,626-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 11:59:35,568-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 11:59:42,774-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 11:59:45,577-0600 INFO Multiplexer Avg stream buf: 0
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-115745.981, now=131106-115946.839, channel=TCPChannel [type: server, contact: 127.0.0.1:36992]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 11:59:46,839-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-115745.981, now=131106-115946.839, channel=TCPChannel [type: server, contact: 127.0.0.1:36992]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 11:59:46,840-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-115745.981, now=131106-115946.839, channel=TCPChannel [type: server, contact: 127.0.0.1:36992]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 11:59:46,840-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-115745.981, now=131106-115946.839, channel=TCPChannel [type: server, contact: 127.0.0.1:36992]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 11:59:46,840-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-115745.981, now=131106-115946.839, channel=TCPChannel [type: server, contact: 127.0.0.1:36992]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 11:59:46,840-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-115745.981, now=131106-115946.839, channel=TCPChannel [type: server, contact: 127.0.0.1:36992]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 11:59:46,840-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-115745.981, now=131106-115946.839, channel=TCPChannel [type: server, contact: 127.0.0.1:36992]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 11:59:46,840-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-115745.981, now=131106-115946.839, channel=TCPChannel [type: server, contact: 127.0.0.1:36992]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 11:59:46,841-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-115745.981, now=131106-115946.839, channel=TCPChannel [type: server, contact: 127.0.0.1:36992]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 11:59:46,841-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-115745.981, now=131106-115946.839, channel=TCPChannel [type: server, contact: 127.0.0.1:36992]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 11:59:46,841-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:36992] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-115745.981, now=131106-115946.839, channel=TCPChannel [type: server, contact: 127.0.0.1:36992]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 11:59:46,841-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:36992] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-115745.981, now=131106-115946.839, channel=TCPChannel [type: server, contact: 127.0.0.1:36992]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-null: Channel shut down
-2013-11-06 11:59:46,841-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-36992, boundTo: TCPChannel [type: server, contact: 127.0.0.1:36992]] binding to NullChannel
-2013-11-06 11:59:46,841-0600 INFO MetaChannel MetaChannel [context: cps-36992, boundTo: TCPChannel [type: server, contact: 127.0.0.1:36992]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:36992]
-2013-11-06 11:59:46,842-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:36992]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-115745.981, now=131106-115946.839, channel=TCPChannel [type: server, contact: 127.0.0.1:36992]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 11:59:46,842-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-115745.981, now=131106-115946.839, channel=TCPChannel [type: server, contact: 127.0.0.1:36992]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 11:59:46,842-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-115745.981, now=131106-115946.839, channel=TCPChannel [type: server, contact: 127.0.0.1:36992]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 11:59:46,842-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-115745.981, now=131106-115946.839, channel=TCPChannel [type: server, contact: 127.0.0.1:36992]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 11:59:46,842-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-115745.981, now=131106-115946.839, channel=TCPChannel [type: server, contact: 127.0.0.1:36992]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 11:59:46,842-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-115745.981, now=131106-115946.839, channel=TCPChannel [type: server, contact: 127.0.0.1:36992]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 11:59:46,842-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-115745.981, now=131106-115946.839, channel=TCPChannel [type: server, contact: 127.0.0.1:36992]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 11:59:46,843-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-115745.981, now=131106-115946.839, channel=TCPChannel [type: server, contact: 127.0.0.1:36992]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 11:59:46,843-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-115745.981, now=131106-115946.839, channel=TCPChannel [type: server, contact: 127.0.0.1:36992]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 11:59:46,843-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-115745.981, now=131106-115946.839, channel=TCPChannel [type: server, contact: 127.0.0.1:36992]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel exception handled
-2013-11-06 11:59:46,843-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 11:59:46,860-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 11:59:46,860-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 11:59:46,861-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:36992] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 11:59:46,861-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:36992] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 11:59:46,861-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:36992]
-2013-11-06 11:59:46,861-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:36992]
-No streams
-2013-11-06 11:59:55,585-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:00:03,100-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:00:05,593-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:00:15,601-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:00:20,092-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:00:25,609-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:00:35,617-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:00:39,574-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 12:00:42,837-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:36993] for addition
-2013-11-06 12:00:42,838-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:36993] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:36993]
-2013-11-06 12:00:42,838-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:36993]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7ff7
-2013-11-06 12:00:42,878-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7ff7
-MetaChannel [context: cps-36993, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:36993]
-2013-11-06 12:00:42,879-0600 INFO MetaChannel MetaChannel [context: cps-36993, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:36993]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 12:00:42,879-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 12:00:42,899-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 12:00:42,899-0600 INFO Sender Sender TCPChannel queue size: 0
-org.globus.cog.coaster.ProtocolException: Invalid line:
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler$Helper.scan(SubmitJobHandler.java:312)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler$Helper.read(SubmitJobHandler.java:270)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.read(SubmitJobHandler.java:97)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.requestComplete(SubmitJobHandler.java:61)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:173)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Handler(tag: 38908, SUBMITJOB) sending error: Could not deserialize job description
-org.globus.cog.coaster.ProtocolException: Could not deserialize job description
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.requestComplete(SubmitJobHandler.java:71)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:173)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Caused by: org.globus.cog.coaster.ProtocolException: Invalid line:
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler$Helper.scan(SubmitJobHandler.java:312)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler$Helper.read(SubmitJobHandler.java:270)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.read(SubmitJobHandler.java:97)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.requestComplete(SubmitJobHandler.java:61)
- ... 4 more
-2013-11-06 12:00:42,980-0600 INFO RequestReply Handler(tag: 38908, SUBMITJOB) sending error: Could not deserialize job description
-org.globus.cog.coaster.ProtocolException: Could not deserialize job description
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.requestComplete(SubmitJobHandler.java:71)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:173)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Caused by: org.globus.cog.coaster.ProtocolException: Invalid line:
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler$Helper.scan(SubmitJobHandler.java:312)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler$Helper.read(SubmitJobHandler.java:270)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.read(SubmitJobHandler.java:97)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.requestComplete(SubmitJobHandler.java:61)
- ... 4 more
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 12:00:42,981-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-Avg stream buf: 0
-2013-11-06 12:00:45,629-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 12:00:55,638-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:00:57,410-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 12:01:05,646-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 12:01:15,655-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:01:16,049-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 12:01:25,664-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:01:34,728-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 12:01:35,610-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:36994] for addition
-2013-11-06 12:01:35,610-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:36994] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:36994]
-2013-11-06 12:01:35,610-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:36994]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7ff6
-2013-11-06 12:01:35,640-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7ff6
-MetaChannel [context: cps-36994, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:36994]
-2013-11-06 12:01:35,640-0600 INFO MetaChannel MetaChannel [context: cps-36994, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:36994]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 12:01:35,640-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Avg stream buf: 27
-2013-11-06 12:01:35,681-0600 INFO Multiplexer Avg stream buf: 27
-Sender TCPChannel queue size: 0
-2013-11-06 12:01:35,681-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 12:01:35,681-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 12:01:35,762-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 12:01:35,777-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 3.0) submitting urn:job-0-1383759274486-1383759274487: /bin/echo
-2013-11-06 12:01:35,777-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 3.0) submitting urn:job-0-1383759274486-1383759274487: /bin/echo
-2013-11-06 12:01:35,777-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 12:01:35,778-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-06 12:01:35,785-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7338, JOBSTATUS) unregistering (send)
-2013-11-06 12:01:35,785-0600 INFO RequestHandler Handler(tag: 7338, JOBSTATUS) unregistering (send)
-2013-11-06 12:01:35,788-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 12:01:35,788-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 12:01:35,788-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-06 12:01:35,788-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7339, JOBSTATUS) unregistering (send)
-2013-11-06 12:01:35,788-0600 INFO RequestHandler Handler(tag: 7339, JOBSTATUS) unregistering (send)
-2013-11-06 12:01:35,788-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-Avg stream buf: 0
-2013-11-06 12:01:45,691-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:01:52,523-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 12:01:55,699-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 12:02:05,708-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:02:12,046-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 12:02:15,716-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 12:02:25,724-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:02:28,997-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 12:02:35,732-0600 INFO Multiplexer Avg stream buf: 0
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 12:02:39,504-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:36995] for addition
-2013-11-06 12:02:39,504-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:36995] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:36995]
-2013-11-06 12:02:39,504-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:36995]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7ff5
-2013-11-06 12:02:39,544-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7ff5
-MetaChannel [context: cps-36995, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:36995]
-2013-11-06 12:02:39,544-0600 INFO MetaChannel MetaChannel [context: cps-36995, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:36995]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 12:02:39,544-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 12:02:39,564-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 12:02:39,564-0600 INFO Sender Sender TCPChannel queue size: 0
-org.globus.cog.coaster.ProtocolException: Invalid line:
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler$Helper.scan(SubmitJobHandler.java:312)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler$Helper.read(SubmitJobHandler.java:270)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.read(SubmitJobHandler.java:95)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.requestComplete(SubmitJobHandler.java:61)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:173)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Handler(tag: 38908, SUBMITJOB) sending error: Could not deserialize job description
-org.globus.cog.coaster.ProtocolException: Could not deserialize job description
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.requestComplete(SubmitJobHandler.java:71)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:173)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Caused by: org.globus.cog.coaster.ProtocolException: Invalid line:
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler$Helper.scan(SubmitJobHandler.java:312)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler$Helper.read(SubmitJobHandler.java:270)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.read(SubmitJobHandler.java:95)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.requestComplete(SubmitJobHandler.java:61)
- ... 4 more
-2013-11-06 12:02:39,645-0600 INFO RequestReply Handler(tag: 38908, SUBMITJOB) sending error: Could not deserialize job description
-org.globus.cog.coaster.ProtocolException: Could not deserialize job description
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.requestComplete(SubmitJobHandler.java:71)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:173)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Caused by: org.globus.cog.coaster.ProtocolException: Invalid line:
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler$Helper.scan(SubmitJobHandler.java:312)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler$Helper.read(SubmitJobHandler.java:270)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.read(SubmitJobHandler.java:95)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.requestComplete(SubmitJobHandler.java:61)
- ... 4 more
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 12:02:39,646-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120042.980, now=131106-120243.838, channel=TCPChannel [type: server, contact: 127.0.0.1:36993]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:02:43,838-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120042.980, now=131106-120243.838, channel=TCPChannel [type: server, contact: 127.0.0.1:36993]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 12:02:43,839-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:36993] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120042.980, now=131106-120243.838, channel=TCPChannel [type: server, contact: 127.0.0.1:36993]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:02:43,839-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:36993] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120042.980, now=131106-120243.838, channel=TCPChannel [type: server, contact: 127.0.0.1:36993]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-null: Channel shut down
-2013-11-06 12:02:43,839-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-36993, boundTo: TCPChannel [type: server, contact: 127.0.0.1:36993]] binding to NullChannel
-2013-11-06 12:02:43,839-0600 INFO MetaChannel MetaChannel [context: cps-36993, boundTo: TCPChannel [type: server, contact: 127.0.0.1:36993]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:36993]
-2013-11-06 12:02:43,840-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:36993]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120042.980, now=131106-120243.838, channel=TCPChannel [type: server, contact: 127.0.0.1:36993]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:02:43,840-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120042.980, now=131106-120243.838, channel=TCPChannel [type: server, contact: 127.0.0.1:36993]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel exception handled
-2013-11-06 12:02:43,840-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 12:02:43,858-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 12:02:43,859-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 12:02:43,859-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:36993] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 12:02:43,859-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:36993] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 12:02:43,859-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:36993]
-2013-11-06 12:02:43,859-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:36993]
-Avg stream buf: 0
-2013-11-06 12:02:45,745-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:02:49,364-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 12:02:55,753-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 2ms, rtt: 4ms
-2013-11-06 12:03:05,474-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 2ms, rtt: 4ms
-Avg stream buf: 0
-2013-11-06 12:03:05,762-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 12:03:15,770-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 12:03:25,779-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 12:03:26,683-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 12:03:35,789-0600 INFO Multiplexer Avg stream buf: 0
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120135.761, now=131106-120336.610, channel=TCPChannel [type: server, contact: 127.0.0.1:36994]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:03:36,610-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120135.761, now=131106-120336.610, channel=TCPChannel [type: server, contact: 127.0.0.1:36994]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 12:03:36,611-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120135.761, now=131106-120336.610, channel=TCPChannel [type: server, contact: 127.0.0.1:36994]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:03:36,611-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120135.761, now=131106-120336.610, channel=TCPChannel [type: server, contact: 127.0.0.1:36994]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 12:03:36,611-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120135.761, now=131106-120336.610, channel=TCPChannel [type: server, contact: 127.0.0.1:36994]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:03:36,611-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120135.761, now=131106-120336.610, channel=TCPChannel [type: server, contact: 127.0.0.1:36994]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 12:03:36,612-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120135.761, now=131106-120336.610, channel=TCPChannel [type: server, contact: 127.0.0.1:36994]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:03:36,612-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120135.761, now=131106-120336.610, channel=TCPChannel [type: server, contact: 127.0.0.1:36994]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 12:03:36,612-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120135.761, now=131106-120336.610, channel=TCPChannel [type: server, contact: 127.0.0.1:36994]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:03:36,612-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120135.761, now=131106-120336.610, channel=TCPChannel [type: server, contact: 127.0.0.1:36994]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 12:03:36,612-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:36994] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120135.761, now=131106-120336.610, channel=TCPChannel [type: server, contact: 127.0.0.1:36994]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:03:36,612-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:36994] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120135.761, now=131106-120336.610, channel=TCPChannel [type: server, contact: 127.0.0.1:36994]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-null: Channel shut down
-2013-11-06 12:03:36,613-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-36994, boundTo: TCPChannel [type: server, contact: 127.0.0.1:36994]] binding to NullChannel
-2013-11-06 12:03:36,613-0600 INFO MetaChannel MetaChannel [context: cps-36994, boundTo: TCPChannel [type: server, contact: 127.0.0.1:36994]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:36994]
-2013-11-06 12:03:36,613-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:36994]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120135.761, now=131106-120336.610, channel=TCPChannel [type: server, contact: 127.0.0.1:36994]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:03:36,613-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120135.761, now=131106-120336.610, channel=TCPChannel [type: server, contact: 127.0.0.1:36994]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 12:03:36,613-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120135.761, now=131106-120336.610, channel=TCPChannel [type: server, contact: 127.0.0.1:36994]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:03:36,613-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120135.761, now=131106-120336.610, channel=TCPChannel [type: server, contact: 127.0.0.1:36994]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 12:03:36,613-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120135.761, now=131106-120336.610, channel=TCPChannel [type: server, contact: 127.0.0.1:36994]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:03:36,613-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120135.761, now=131106-120336.610, channel=TCPChannel [type: server, contact: 127.0.0.1:36994]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 12:03:36,614-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120135.761, now=131106-120336.610, channel=TCPChannel [type: server, contact: 127.0.0.1:36994]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:03:36,614-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120135.761, now=131106-120336.610, channel=TCPChannel [type: server, contact: 127.0.0.1:36994]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 12:03:36,614-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120135.761, now=131106-120336.610, channel=TCPChannel [type: server, contact: 127.0.0.1:36994]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:03:36,614-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120135.761, now=131106-120336.610, channel=TCPChannel [type: server, contact: 127.0.0.1:36994]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel exception handled
-2013-11-06 12:03:36,614-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 12:03:36,631-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 12:03:36,631-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 12:03:36,632-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:36994] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 12:03:36,632-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:36994] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 12:03:36,632-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:36994]
-2013-11-06 12:03:36,632-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:36994]
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 12:03:41,946-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-Avg stream buf: 0
-2013-11-06 12:03:45,799-0600 INFO Multiplexer Avg stream buf: 0
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 12:03:48,640-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:36999] for addition
-2013-11-06 12:03:48,641-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:36999] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:36999]
-2013-11-06 12:03:48,641-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:36999]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7ff4
-2013-11-06 12:03:48,668-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7ff4
-MetaChannel [context: cps-36999, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:36999]
-2013-11-06 12:03:48,668-0600 INFO MetaChannel MetaChannel [context: cps-36999, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:36999]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 12:03:48,668-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 12:03:48,708-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 12:03:48,708-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 12:03:48,789-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 12:03:48,790-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 4.0) submitting urn:job-0-1383759274492-1383759274493: /bin/date
-2013-11-06 12:03:48,790-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 4.0) submitting urn:job-0-1383759274492-1383759274493: /bin/date
-2013-11-06 12:03:48,791-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 12:03:48,791-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-06 12:03:48,797-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7340, JOBSTATUS) unregistering (send)
-2013-11-06 12:03:48,797-0600 INFO RequestHandler Handler(tag: 7340, JOBSTATUS) unregistering (send)
-2013-11-06 12:03:48,809-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 12:03:48,809-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 12:03:48,809-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-06 12:03:48,809-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7341, JOBSTATUS) unregistering (send)
-2013-11-06 12:03:48,809-0600 INFO RequestHandler Handler(tag: 7341, JOBSTATUS) unregistering (send)
-2013-11-06 12:03:48,809-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-Avg stream buf: 0
-2013-11-06 12:03:55,809-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 12:04:04,001-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 12:04:05,818-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 12:04:15,827-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:04:18,419-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 12:04:25,836-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 12:04:35,845-0600 INFO Multiplexer Avg stream buf: 0
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120239.645, now=131106-120440.504, channel=TCPChannel [type: server, contact: 127.0.0.1:36995]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:04:40,504-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120239.645, now=131106-120440.504, channel=TCPChannel [type: server, contact: 127.0.0.1:36995]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 12:04:40,504-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:36995] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120239.645, now=131106-120440.504, channel=TCPChannel [type: server, contact: 127.0.0.1:36995]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:04:40,504-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:36995] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120239.645, now=131106-120440.504, channel=TCPChannel [type: server, contact: 127.0.0.1:36995]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-null: Channel shut down
-2013-11-06 12:04:40,505-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-36995, boundTo: TCPChannel [type: server, contact: 127.0.0.1:36995]] binding to NullChannel
-2013-11-06 12:04:40,505-0600 INFO MetaChannel MetaChannel [context: cps-36995, boundTo: TCPChannel [type: server, contact: 127.0.0.1:36995]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:36995]
-2013-11-06 12:04:40,505-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:36995]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120239.645, now=131106-120440.504, channel=TCPChannel [type: server, contact: 127.0.0.1:36995]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:04:40,505-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120239.645, now=131106-120440.504, channel=TCPChannel [type: server, contact: 127.0.0.1:36995]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel exception handled
-2013-11-06 12:04:40,505-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 12:04:40,519-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 12:04:40,519-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 12:04:40,519-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:36995] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 12:04:40,519-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:36995] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 12:04:40,519-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:36995]
-2013-11-06 12:04:40,519-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:36995]
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:04:41,319-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 12:04:45,855-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:04:54,893-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 12:04:55,864-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 12:05:05,873-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 12:05:15,881-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 12:05:18,638-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 12:05:25,890-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 12:05:31,368-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-Avg stream buf: 0
-2013-11-06 12:05:35,899-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 12:05:45,908-0600 INFO Multiplexer Avg stream buf: 0
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120348.789, now=131106-120549.642, channel=TCPChannel [type: server, contact: 127.0.0.1:36999]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:05:49,642-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120348.789, now=131106-120549.642, channel=TCPChannel [type: server, contact: 127.0.0.1:36999]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 12:05:49,642-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120348.789, now=131106-120549.642, channel=TCPChannel [type: server, contact: 127.0.0.1:36999]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:05:49,642-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120348.789, now=131106-120549.642, channel=TCPChannel [type: server, contact: 127.0.0.1:36999]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 12:05:49,643-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120348.789, now=131106-120549.642, channel=TCPChannel [type: server, contact: 127.0.0.1:36999]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:05:49,643-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120348.789, now=131106-120549.642, channel=TCPChannel [type: server, contact: 127.0.0.1:36999]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 12:05:49,643-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120348.789, now=131106-120549.642, channel=TCPChannel [type: server, contact: 127.0.0.1:36999]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:05:49,643-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120348.789, now=131106-120549.642, channel=TCPChannel [type: server, contact: 127.0.0.1:36999]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 12:05:49,643-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120348.789, now=131106-120549.642, channel=TCPChannel [type: server, contact: 127.0.0.1:36999]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:05:49,643-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120348.789, now=131106-120549.642, channel=TCPChannel [type: server, contact: 127.0.0.1:36999]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 12:05:49,643-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:36999] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120348.789, now=131106-120549.642, channel=TCPChannel [type: server, contact: 127.0.0.1:36999]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:05:49,644-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:36999] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120348.789, now=131106-120549.642, channel=TCPChannel [type: server, contact: 127.0.0.1:36999]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-null: Channel shut down
-2013-11-06 12:05:49,644-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-36999, boundTo: TCPChannel [type: server, contact: 127.0.0.1:36999]] binding to NullChannel
-2013-11-06 12:05:49,644-0600 INFO MetaChannel MetaChannel [context: cps-36999, boundTo: TCPChannel [type: server, contact: 127.0.0.1:36999]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:36999]
-2013-11-06 12:05:49,644-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:36999]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120348.789, now=131106-120549.642, channel=TCPChannel [type: server, contact: 127.0.0.1:36999]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:05:49,644-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120348.789, now=131106-120549.642, channel=TCPChannel [type: server, contact: 127.0.0.1:36999]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 12:05:49,644-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120348.789, now=131106-120549.642, channel=TCPChannel [type: server, contact: 127.0.0.1:36999]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:05:49,645-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120348.789, now=131106-120549.642, channel=TCPChannel [type: server, contact: 127.0.0.1:36999]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 12:05:49,645-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120348.789, now=131106-120549.642, channel=TCPChannel [type: server, contact: 127.0.0.1:36999]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:05:49,645-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120348.789, now=131106-120549.642, channel=TCPChannel [type: server, contact: 127.0.0.1:36999]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 12:05:49,645-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120348.789, now=131106-120549.642, channel=TCPChannel [type: server, contact: 127.0.0.1:36999]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:05:49,645-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120348.789, now=131106-120549.642, channel=TCPChannel [type: server, contact: 127.0.0.1:36999]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 12:05:49,645-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120348.789, now=131106-120549.642, channel=TCPChannel [type: server, contact: 127.0.0.1:36999]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:05:49,645-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120348.789, now=131106-120549.642, channel=TCPChannel [type: server, contact: 127.0.0.1:36999]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel exception handled
-2013-11-06 12:05:49,645-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 12:05:49,659-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 12:05:49,659-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 12:05:49,659-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:36999] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 12:05:49,659-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:36999] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 12:05:49,659-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:36999]
-2013-11-06 12:05:49,660-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:36999]
-No streams
-2013-11-06 12:05:55,917-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:05:55,955-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:06:05,925-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:06:07,841-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:06:15,934-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:06:25,942-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:06:33,273-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:06:35,950-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:06:44,315-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:06:45,959-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:06:55,967-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:07:05,976-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:07:10,592-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:07:15,984-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:07:20,789-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:07:25,992-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:07:36,000-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:07:46,009-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:07:47,909-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:07:56,017-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 12:07:57,264-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 12:08:05,142-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:37005] for addition
-2013-11-06 12:08:05,143-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:37005] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:37005]
-2013-11-06 12:08:05,143-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:37005]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7ff3
-2013-11-06 12:08:05,164-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7ff3
-MetaChannel [context: cps-37005, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:37005]
-2013-11-06 12:08:05,164-0600 INFO MetaChannel MetaChannel [context: cps-37005, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:37005]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 12:08:05,164-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 12:08:05,204-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 12:08:05,204-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 12:08:05,285-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 12:08:05,302-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 5.0) submitting urn:job-0-1383759274495-1383759274496: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 12:08:05,303-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 5.0) submitting urn:job-0-1383759274495-1383759274496: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 12:08:05,303-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 12:08:05,303-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-06 12:08:05,309-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7342, JOBSTATUS) unregistering (send)
-2013-11-06 12:08:05,309-0600 INFO RequestHandler Handler(tag: 7342, JOBSTATUS) unregistering (send)
-Avg stream buf: 0
-2013-11-06 12:08:06,027-0600 INFO Multiplexer Avg stream buf: 0
-2013-11-06 12:08:07,535-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 12:08:07,535-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 12:08:07,536-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-06 12:08:07,536-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7343, JOBSTATUS) unregistering (send)
-2013-11-06 12:08:07,536-0600 INFO RequestHandler Handler(tag: 7343, JOBSTATUS) unregistering (send)
-2013-11-06 12:08:07,536-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-Avg stream buf: 0
-2013-11-06 12:08:16,035-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:08:25,227-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 12:08:26,042-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:08:33,738-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 12:08:36,050-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 12:08:46,058-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 12:08:56,066-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:09:02,545-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 12:09:06,073-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:09:10,212-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 12:09:16,081-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 12:09:26,089-0600 INFO Multiplexer Avg stream buf: 0
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 12:09:27,301-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:37006] for addition
-2013-11-06 12:09:27,302-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:37006] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:37006]
-2013-11-06 12:09:27,302-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:37006]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7ff2
-2013-11-06 12:09:27,334-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7ff2
-MetaChannel [context: cps-37006, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:37006]
-2013-11-06 12:09:27,334-0600 INFO MetaChannel MetaChannel [context: cps-37006, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:37006]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 12:09:27,334-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 12:09:27,395-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 12:09:27,395-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 12:09:27,475-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 12:09:27,480-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 6.0) submitting urn:job-0-1383759274498-1383759274499: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 12:09:27,480-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 6.0) submitting urn:job-0-1383759274498-1383759274499: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 12:09:27,480-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 12:09:27,480-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-06 12:09:27,486-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7344, JOBSTATUS) unregistering (send)
-2013-11-06 12:09:27,487-0600 INFO RequestHandler Handler(tag: 7344, JOBSTATUS) unregistering (send)
-2013-11-06 12:09:29,508-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 12:09:29,508-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 12:09:29,508-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-06 12:09:29,508-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7345, JOBSTATUS) unregistering (send)
-2013-11-06 12:09:29,508-0600 INFO RequestHandler Handler(tag: 7345, JOBSTATUS) unregistering (send)
-2013-11-06 12:09:29,508-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-Avg stream buf: 0
-2013-11-06 12:09:36,100-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 12:09:39,863-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 12:09:46,109-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:09:46,687-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 12:09:56,117-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 12:10:06,126-0600 INFO Multiplexer Avg stream buf: 0
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120805.285, now=131106-121006.143, channel=TCPChannel [type: server, contact: 127.0.0.1:37005]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:10:06,143-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120805.285, now=131106-121006.143, channel=TCPChannel [type: server, contact: 127.0.0.1:37005]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 12:10:06,143-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120805.285, now=131106-121006.143, channel=TCPChannel [type: server, contact: 127.0.0.1:37005]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:10:06,144-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120805.285, now=131106-121006.143, channel=TCPChannel [type: server, contact: 127.0.0.1:37005]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 12:10:06,144-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120805.285, now=131106-121006.143, channel=TCPChannel [type: server, contact: 127.0.0.1:37005]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:10:06,144-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120805.285, now=131106-121006.143, channel=TCPChannel [type: server, contact: 127.0.0.1:37005]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 12:10:06,144-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120805.285, now=131106-121006.143, channel=TCPChannel [type: server, contact: 127.0.0.1:37005]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:10:06,144-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120805.285, now=131106-121006.143, channel=TCPChannel [type: server, contact: 127.0.0.1:37005]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 12:10:06,144-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120805.285, now=131106-121006.143, channel=TCPChannel [type: server, contact: 127.0.0.1:37005]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:10:06,145-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120805.285, now=131106-121006.143, channel=TCPChannel [type: server, contact: 127.0.0.1:37005]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 12:10:06,145-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:37005] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120805.285, now=131106-121006.143, channel=TCPChannel [type: server, contact: 127.0.0.1:37005]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:10:06,145-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:37005] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120805.285, now=131106-121006.143, channel=TCPChannel [type: server, contact: 127.0.0.1:37005]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-null: Channel shut down
-2013-11-06 12:10:06,145-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-37005, boundTo: TCPChannel [type: server, contact: 127.0.0.1:37005]] binding to NullChannel
-2013-11-06 12:10:06,145-0600 INFO MetaChannel MetaChannel [context: cps-37005, boundTo: TCPChannel [type: server, contact: 127.0.0.1:37005]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:37005]
-2013-11-06 12:10:06,146-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:37005]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120805.285, now=131106-121006.143, channel=TCPChannel [type: server, contact: 127.0.0.1:37005]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:10:06,146-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120805.285, now=131106-121006.143, channel=TCPChannel [type: server, contact: 127.0.0.1:37005]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel is closed. Removing.
-2013-11-06 12:10:06,146-0600 INFO Multiplexer Channel is closed. Removing.
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 12:10:06,146-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 12:10:06,146-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120805.285, now=131106-121006.143, channel=TCPChannel [type: server, contact: 127.0.0.1:37005]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:10:06,146-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120805.285, now=131106-121006.143, channel=TCPChannel [type: server, contact: 127.0.0.1:37005]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 12:10:06,146-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120805.285, now=131106-121006.143, channel=TCPChannel [type: server, contact: 127.0.0.1:37005]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:10:06,146-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120805.285, now=131106-121006.143, channel=TCPChannel [type: server, contact: 127.0.0.1:37005]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 12:10:06,147-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120805.285, now=131106-121006.143, channel=TCPChannel [type: server, contact: 127.0.0.1:37005]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:10:06,147-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120805.285, now=131106-121006.143, channel=TCPChannel [type: server, contact: 127.0.0.1:37005]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 12:10:06,147-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120805.285, now=131106-121006.143, channel=TCPChannel [type: server, contact: 127.0.0.1:37005]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:10:06,147-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120805.285, now=131106-121006.143, channel=TCPChannel [type: server, contact: 127.0.0.1:37005]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel exception handled
-2013-11-06 12:10:06,147-0600 INFO ChannelManager Channel exception handled
-Channel config: null -> {keepalive=null}
-2013-11-06 12:10:06,147-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:37005] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 12:10:06,147-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:37005] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 12:10:06,148-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:37005]
-2013-11-06 12:10:06,148-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:37005]
-Avg stream buf: 0
-2013-11-06 12:10:16,136-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:10:17,181-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 12:10:23,161-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-Avg stream buf: 0
-2013-11-06 12:10:26,144-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 12:10:36,152-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 12:10:46,161-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:10:54,499-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 12:10:56,169-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:10:59,634-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 12:11:06,177-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 12:11:16,185-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 12:11:26,194-0600 INFO Multiplexer Avg stream buf: 0
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120927.475, now=131106-121128.302, channel=TCPChannel [type: server, contact: 127.0.0.1:37006]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:11:28,302-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120927.475, now=131106-121128.302, channel=TCPChannel [type: server, contact: 127.0.0.1:37006]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 12:11:28,303-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120927.475, now=131106-121128.302, channel=TCPChannel [type: server, contact: 127.0.0.1:37006]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:11:28,303-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120927.475, now=131106-121128.302, channel=TCPChannel [type: server, contact: 127.0.0.1:37006]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 12:11:28,303-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120927.475, now=131106-121128.302, channel=TCPChannel [type: server, contact: 127.0.0.1:37006]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:11:28,303-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120927.475, now=131106-121128.302, channel=TCPChannel [type: server, contact: 127.0.0.1:37006]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 12:11:28,303-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120927.475, now=131106-121128.302, channel=TCPChannel [type: server, contact: 127.0.0.1:37006]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:11:28,303-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120927.475, now=131106-121128.302, channel=TCPChannel [type: server, contact: 127.0.0.1:37006]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 12:11:28,304-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120927.475, now=131106-121128.302, channel=TCPChannel [type: server, contact: 127.0.0.1:37006]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:11:28,304-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120927.475, now=131106-121128.302, channel=TCPChannel [type: server, contact: 127.0.0.1:37006]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 12:11:28,304-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:37006] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120927.475, now=131106-121128.302, channel=TCPChannel [type: server, contact: 127.0.0.1:37006]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:11:28,304-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:37006] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120927.475, now=131106-121128.302, channel=TCPChannel [type: server, contact: 127.0.0.1:37006]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-null: Channel shut down
-2013-11-06 12:11:28,304-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-37006, boundTo: TCPChannel [type: server, contact: 127.0.0.1:37006]] binding to NullChannel
-2013-11-06 12:11:28,304-0600 INFO MetaChannel MetaChannel [context: cps-37006, boundTo: TCPChannel [type: server, contact: 127.0.0.1:37006]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:37006]
-2013-11-06 12:11:28,305-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:37006]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120927.475, now=131106-121128.302, channel=TCPChannel [type: server, contact: 127.0.0.1:37006]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:11:28,305-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120927.475, now=131106-121128.302, channel=TCPChannel [type: server, contact: 127.0.0.1:37006]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 12:11:28,305-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120927.475, now=131106-121128.302, channel=TCPChannel [type: server, contact: 127.0.0.1:37006]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:11:28,305-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120927.475, now=131106-121128.302, channel=TCPChannel [type: server, contact: 127.0.0.1:37006]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 12:11:28,305-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120927.475, now=131106-121128.302, channel=TCPChannel [type: server, contact: 127.0.0.1:37006]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:11:28,305-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120927.475, now=131106-121128.302, channel=TCPChannel [type: server, contact: 127.0.0.1:37006]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 12:11:28,305-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120927.475, now=131106-121128.302, channel=TCPChannel [type: server, contact: 127.0.0.1:37006]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:11:28,305-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120927.475, now=131106-121128.302, channel=TCPChannel [type: server, contact: 127.0.0.1:37006]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 12:11:28,306-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120927.475, now=131106-121128.302, channel=TCPChannel [type: server, contact: 127.0.0.1:37006]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:11:28,306-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-120927.475, now=131106-121128.302, channel=TCPChannel [type: server, contact: 127.0.0.1:37006]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel exception handled
-2013-11-06 12:11:28,306-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 12:11:28,320-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 12:11:28,320-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 12:11:28,320-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:37006] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 12:11:28,320-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:37006] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 12:11:28,320-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:37006]
-2013-11-06 12:11:28,320-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:37006]
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:11:31,817-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:11:36,108-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:11:36,202-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:11:46,210-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:11:56,217-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:12:06,225-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 12:12:09,135-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 12:12:12,583-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 12:12:16,233-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:12:26,240-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:12:36,248-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:12:46,257-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 12:12:46,453-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 12:12:49,057-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 12:12:56,266-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:13:06,275-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:13:16,289-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 12:13:23,772-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:13:25,530-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:13:26,290-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:13:36,299-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:13:46,308-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:13:56,314-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 12:14:01,090-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:14:02,004-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:14:06,326-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:14:16,339-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:14:26,351-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:14:36,362-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 3ms
-2013-11-06 12:14:38,409-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 3ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:14:38,478-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:14:46,375-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:14:56,389-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:15:06,398-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 12:15:14,953-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 12:15:15,726-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 12:15:16,408-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:15:26,417-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:15:36,427-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:15:46,437-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:15:51,426-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:15:53,043-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:15:56,447-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:16:06,457-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:16:16,467-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:16:26,477-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 12:16:27,901-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:16:30,361-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:16:36,487-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:16:46,497-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:16:56,506-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:17:04,374-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:17:06,516-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 12:17:07,680-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 12:17:16,526-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:17:26,536-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:17:36,545-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 12:17:40,849-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:17:44,997-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:17:46,554-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:17:56,562-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:18:06,569-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:18:16,577-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 12:18:17,323-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:18:22,315-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:18:26,585-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:18:36,593-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:18:46,601-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 12:18:53,797-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 12:18:56,609-0600 INFO Multiplexer No streams
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 12:18:57,254-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:37028] for addition
-2013-11-06 12:18:57,254-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:37028] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:37028]
-2013-11-06 12:18:57,254-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:37028]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7ff1
-2013-11-06 12:18:57,293-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7ff1
-MetaChannel [context: cps-37028, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:37028]
-2013-11-06 12:18:57,293-0600 INFO MetaChannel MetaChannel [context: cps-37028, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:37028]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 12:18:57,293-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 12:18:57,314-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 12:18:57,314-0600 INFO Sender Sender TCPChannel queue size: 0
-org.globus.cog.coaster.ProtocolException: Invalid line:
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler$Helper.scan(SubmitJobHandler.java:312)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler$Helper.read(SubmitJobHandler.java:270)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.read(SubmitJobHandler.java:97)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.requestComplete(SubmitJobHandler.java:61)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:173)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Handler(tag: 38908, SUBMITJOB) sending error: Could not deserialize job description
-org.globus.cog.coaster.ProtocolException: Could not deserialize job description
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.requestComplete(SubmitJobHandler.java:71)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:173)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Caused by: org.globus.cog.coaster.ProtocolException: Invalid line:
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler$Helper.scan(SubmitJobHandler.java:312)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler$Helper.read(SubmitJobHandler.java:270)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.read(SubmitJobHandler.java:97)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.requestComplete(SubmitJobHandler.java:61)
- ... 4 more
-2013-11-06 12:18:57,395-0600 INFO RequestReply Handler(tag: 38908, SUBMITJOB) sending error: Could not deserialize job description
-org.globus.cog.coaster.ProtocolException: Could not deserialize job description
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.requestComplete(SubmitJobHandler.java:71)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:173)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Caused by: org.globus.cog.coaster.ProtocolException: Invalid line:
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler$Helper.scan(SubmitJobHandler.java:312)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler$Helper.read(SubmitJobHandler.java:270)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.read(SubmitJobHandler.java:97)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.requestComplete(SubmitJobHandler.java:61)
- ... 4 more
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 12:18:57,396-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:18:59,634-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 12:19:06,623-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 12:19:16,631-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 12:19:26,640-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:19:30,270-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 12:19:36,649-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 12:19:36,951-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 12:19:45,474-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:37034] for addition
-2013-11-06 12:19:45,474-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:37034] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:37034]
-2013-11-06 12:19:45,474-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:37034]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7ff0
-2013-11-06 12:19:45,495-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7ff0
-MetaChannel [context: cps-37034, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:37034]
-2013-11-06 12:19:45,496-0600 INFO MetaChannel MetaChannel [context: cps-37034, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:37034]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 12:19:45,496-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 12:19:45,537-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 12:19:45,537-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 12:19:45,618-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 12:19:45,657-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 7.0) submitting urn:job-0-1383759274504-1383759274505: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 12:19:45,657-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 7.0) submitting urn:job-0-1383759274504-1383759274505: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 12:19:45,658-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 12:19:45,658-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-06 12:19:45,663-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7346, JOBSTATUS) unregistering (send)
-2013-11-06 12:19:45,663-0600 INFO RequestHandler Handler(tag: 7346, JOBSTATUS) unregistering (send)
-Avg stream buf: 0
-2013-11-06 12:19:46,661-0600 INFO Multiplexer Avg stream buf: 0
-2013-11-06 12:19:47,865-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 12:19:47,865-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 12:19:47,865-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-06 12:19:47,866-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7347, JOBSTATUS) unregistering (send)
-2013-11-06 12:19:47,866-0600 INFO RequestHandler Handler(tag: 7347, JOBSTATUS) unregistering (send)
-2013-11-06 12:19:47,866-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 12:19:47,868-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:37035] for addition
-2013-11-06 12:19:47,868-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:37035] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:37035]
-2013-11-06 12:19:47,868-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:37035]
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7fef
-2013-11-06 12:19:47,905-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7fef
-MetaChannel [context: cps-37035, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:37035]
-2013-11-06 12:19:47,905-0600 INFO MetaChannel MetaChannel [context: cps-37035, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:37035]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-06 12:19:47,905-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-06 12:19:47,945-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 12:19:48,027-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 12:19:48,066-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 8.0) submitting urn:job-1-1383759274507-1383759274508: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 12:19:48,066-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 8.0) submitting urn:job-1-1383759274507-1383759274508: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 12:19:48,067-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 12:19:48,067-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-06 12:19:48,072-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7348, JOBSTATUS) unregistering (send)
-2013-11-06 12:19:48,073-0600 INFO RequestHandler Handler(tag: 7348, JOBSTATUS) unregistering (send)
-2013-11-06 12:19:50,088-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 12:19:50,088-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 12:19:50,088-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-06 12:19:50,088-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7349, JOBSTATUS) unregistering (send)
-2013-11-06 12:19:50,088-0600 INFO RequestHandler Handler(tag: 7349, JOBSTATUS) unregistering (send)
-2013-11-06 12:19:50,088-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 12:19:50,089-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:37036] for addition
-2013-11-06 12:19:50,090-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:37036] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:37036]
-2013-11-06 12:19:50,090-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:37036]
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7fee
-2013-11-06 12:19:50,113-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7fee
-MetaChannel [context: cps-37036, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:37036]
-2013-11-06 12:19:50,113-0600 INFO MetaChannel MetaChannel [context: cps-37036, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:37036]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-06 12:19:50,113-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-Avg stream buf: 0
-2013-11-06 12:19:56,679-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 12:20:06,690-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:20:06,745-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 12:20:14,269-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-Avg stream buf: 0
-2013-11-06 12:20:16,700-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 12:20:26,709-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 12:20:36,719-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:20:43,219-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 12:20:46,729-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 12:20:51,587-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 12:20:56,739-0600 INFO Multiplexer Avg stream buf: 0
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-121857.394, now=131106-122058.254, channel=TCPChannel [type: server, contact: 127.0.0.1:37028]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:20:58,255-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-121857.394, now=131106-122058.254, channel=TCPChannel [type: server, contact: 127.0.0.1:37028]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 12:20:58,255-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:37028] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-121857.394, now=131106-122058.254, channel=TCPChannel [type: server, contact: 127.0.0.1:37028]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:20:58,255-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:37028] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-121857.394, now=131106-122058.254, channel=TCPChannel [type: server, contact: 127.0.0.1:37028]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-null: Channel shut down
-2013-11-06 12:20:58,255-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-37028, boundTo: TCPChannel [type: server, contact: 127.0.0.1:37028]] binding to NullChannel
-2013-11-06 12:20:58,256-0600 INFO MetaChannel MetaChannel [context: cps-37028, boundTo: TCPChannel [type: server, contact: 127.0.0.1:37028]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:37028]
-2013-11-06 12:20:58,256-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:37028]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-121857.394, now=131106-122058.254, channel=TCPChannel [type: server, contact: 127.0.0.1:37028]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:20:58,256-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-121857.394, now=131106-122058.254, channel=TCPChannel [type: server, contact: 127.0.0.1:37028]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel exception handled
-2013-11-06 12:20:58,256-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 12:20:58,264-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 12:20:58,264-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 12:20:58,265-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:37028] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 12:20:58,265-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:37028] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 12:20:58,265-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:37028]
-2013-11-06 12:20:58,265-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:37028]
-Avg stream buf: 0
-2013-11-06 12:21:06,750-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 12:21:16,760-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:21:19,693-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 12:21:26,769-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:21:28,905-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 12:21:36,778-0600 INFO Multiplexer Avg stream buf: 0
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-121945.617, now=131106-122146.475, channel=TCPChannel [type: server, contact: 127.0.0.1:37034]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:21:46,475-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-121945.617, now=131106-122146.475, channel=TCPChannel [type: server, contact: 127.0.0.1:37034]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 12:21:46,475-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-121945.617, now=131106-122146.475, channel=TCPChannel [type: server, contact: 127.0.0.1:37034]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:21:46,475-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-121945.617, now=131106-122146.475, channel=TCPChannel [type: server, contact: 127.0.0.1:37034]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 12:21:46,475-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-121945.617, now=131106-122146.475, channel=TCPChannel [type: server, contact: 127.0.0.1:37034]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:21:46,475-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-121945.617, now=131106-122146.475, channel=TCPChannel [type: server, contact: 127.0.0.1:37034]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 12:21:46,476-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-121945.617, now=131106-122146.475, channel=TCPChannel [type: server, contact: 127.0.0.1:37034]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:21:46,476-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-121945.617, now=131106-122146.475, channel=TCPChannel [type: server, contact: 127.0.0.1:37034]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 12:21:46,476-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-121945.617, now=131106-122146.475, channel=TCPChannel [type: server, contact: 127.0.0.1:37034]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:21:46,476-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-121945.617, now=131106-122146.475, channel=TCPChannel [type: server, contact: 127.0.0.1:37034]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 12:21:46,476-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:37034] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-121945.617, now=131106-122146.475, channel=TCPChannel [type: server, contact: 127.0.0.1:37034]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:21:46,476-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:37034] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-121945.617, now=131106-122146.475, channel=TCPChannel [type: server, contact: 127.0.0.1:37034]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-null: Channel shut down
-2013-11-06 12:21:46,477-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-37034, boundTo: TCPChannel [type: server, contact: 127.0.0.1:37034]] binding to NullChannel
-2013-11-06 12:21:46,477-0600 INFO MetaChannel MetaChannel [context: cps-37034, boundTo: TCPChannel [type: server, contact: 127.0.0.1:37034]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:37034]
-2013-11-06 12:21:46,477-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:37034]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-121945.617, now=131106-122146.475, channel=TCPChannel [type: server, contact: 127.0.0.1:37034]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:21:46,477-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-121945.617, now=131106-122146.475, channel=TCPChannel [type: server, contact: 127.0.0.1:37034]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 12:21:46,477-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-121945.617, now=131106-122146.475, channel=TCPChannel [type: server, contact: 127.0.0.1:37034]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:21:46,477-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-121945.617, now=131106-122146.475, channel=TCPChannel [type: server, contact: 127.0.0.1:37034]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 12:21:46,477-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-121945.617, now=131106-122146.475, channel=TCPChannel [type: server, contact: 127.0.0.1:37034]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:21:46,477-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-121945.617, now=131106-122146.475, channel=TCPChannel [type: server, contact: 127.0.0.1:37034]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 12:21:46,478-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-121945.617, now=131106-122146.475, channel=TCPChannel [type: server, contact: 127.0.0.1:37034]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:21:46,478-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-121945.617, now=131106-122146.475, channel=TCPChannel [type: server, contact: 127.0.0.1:37034]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 12:21:46,478-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-121945.617, now=131106-122146.475, channel=TCPChannel [type: server, contact: 127.0.0.1:37034]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:21:46,478-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-121945.617, now=131106-122146.475, channel=TCPChannel [type: server, contact: 127.0.0.1:37034]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel exception handled
-2013-11-06 12:21:46,478-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 12:21:46,486-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 12:21:46,486-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 12:21:46,486-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:37034] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 12:21:46,486-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:37034] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 12:21:46,486-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:37034]
-2013-11-06 12:21:46,486-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:37034]
-Avg stream buf: 0
-2013-11-06 12:21:46,787-0600 INFO Multiplexer Avg stream buf: 0
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-121948.026, now=131106-122148.868, channel=TCPChannel [type: server, contact: 127.0.0.1:37035]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:21:48,868-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-121948.026, now=131106-122148.868, channel=TCPChannel [type: server, contact: 127.0.0.1:37035]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 12:21:48,868-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-121948.026, now=131106-122148.868, channel=TCPChannel [type: server, contact: 127.0.0.1:37035]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:21:48,868-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-121948.026, now=131106-122148.868, channel=TCPChannel [type: server, contact: 127.0.0.1:37035]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 12:21:48,868-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-121948.026, now=131106-122148.868, channel=TCPChannel [type: server, contact: 127.0.0.1:37035]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:21:48,869-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-121948.026, now=131106-122148.868, channel=TCPChannel [type: server, contact: 127.0.0.1:37035]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 12:21:48,869-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-121948.026, now=131106-122148.868, channel=TCPChannel [type: server, contact: 127.0.0.1:37035]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:21:48,869-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-121948.026, now=131106-122148.868, channel=TCPChannel [type: server, contact: 127.0.0.1:37035]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 12:21:48,869-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-121948.026, now=131106-122148.868, channel=TCPChannel [type: server, contact: 127.0.0.1:37035]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:21:48,869-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-121948.026, now=131106-122148.868, channel=TCPChannel [type: server, contact: 127.0.0.1:37035]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 12:21:48,870-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:37035] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-121948.026, now=131106-122148.868, channel=TCPChannel [type: server, contact: 127.0.0.1:37035]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:21:48,870-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:37035] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-121948.026, now=131106-122148.868, channel=TCPChannel [type: server, contact: 127.0.0.1:37035]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-null: Channel shut down
-2013-11-06 12:21:48,870-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-37035, boundTo: TCPChannel [type: server, contact: 127.0.0.1:37035]] binding to NullChannel
-2013-11-06 12:21:48,870-0600 INFO MetaChannel MetaChannel [context: cps-37035, boundTo: TCPChannel [type: server, contact: 127.0.0.1:37035]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:37035]
-2013-11-06 12:21:48,870-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:37035]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-121948.026, now=131106-122148.868, channel=TCPChannel [type: server, contact: 127.0.0.1:37035]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:21:48,870-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-121948.026, now=131106-122148.868, channel=TCPChannel [type: server, contact: 127.0.0.1:37035]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 12:21:48,870-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-121948.026, now=131106-122148.868, channel=TCPChannel [type: server, contact: 127.0.0.1:37035]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:21:48,870-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-121948.026, now=131106-122148.868, channel=TCPChannel [type: server, contact: 127.0.0.1:37035]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 12:21:48,871-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-121948.026, now=131106-122148.868, channel=TCPChannel [type: server, contact: 127.0.0.1:37035]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:21:48,871-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-121948.026, now=131106-122148.868, channel=TCPChannel [type: server, contact: 127.0.0.1:37035]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 12:21:48,871-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-121948.026, now=131106-122148.868, channel=TCPChannel [type: server, contact: 127.0.0.1:37035]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:21:48,871-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-121948.026, now=131106-122148.868, channel=TCPChannel [type: server, contact: 127.0.0.1:37035]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 12:21:48,871-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-121948.026, now=131106-122148.868, channel=TCPChannel [type: server, contact: 127.0.0.1:37035]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:21:48,871-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-121948.026, now=131106-122148.868, channel=TCPChannel [type: server, contact: 127.0.0.1:37035]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel exception handled
-2013-11-06 12:21:48,871-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 12:21:48,873-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 12:21:48,873-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 12:21:48,874-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:37035] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 12:21:48,874-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:37035] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 12:21:48,874-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:37035]
-2013-11-06 12:21:48,874-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:37035]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-121950.113, now=131106-122151.089, channel=TCPChannel [type: server, contact: 127.0.0.1:37036]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:21:51,089-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-121950.113, now=131106-122151.089, channel=TCPChannel [type: server, contact: 127.0.0.1:37036]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 12:21:51,089-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:37036] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-121950.113, now=131106-122151.089, channel=TCPChannel [type: server, contact: 127.0.0.1:37036]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:21:51,089-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:37036] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-121950.113, now=131106-122151.089, channel=TCPChannel [type: server, contact: 127.0.0.1:37036]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-null: Channel shut down
-2013-11-06 12:21:51,090-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-37036, boundTo: TCPChannel [type: server, contact: 127.0.0.1:37036]] binding to NullChannel
-2013-11-06 12:21:51,090-0600 INFO MetaChannel MetaChannel [context: cps-37036, boundTo: TCPChannel [type: server, contact: 127.0.0.1:37036]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:37036]
-2013-11-06 12:21:51,090-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:37036]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-121950.113, now=131106-122151.089, channel=TCPChannel [type: server, contact: 127.0.0.1:37036]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:21:51,090-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-121950.113, now=131106-122151.089, channel=TCPChannel [type: server, contact: 127.0.0.1:37036]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel exception handled
-2013-11-06 12:21:51,090-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 12:21:51,100-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 12:21:51,100-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 12:21:51,101-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:37036] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 12:21:51,101-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:37036] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 12:21:51,101-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:37036]
-2013-11-06 12:21:51,101-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:37036]
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:21:56,167-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:21:56,797-0600 INFO Multiplexer No streams
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 12:22:01,121-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:37037] for addition
-2013-11-06 12:22:01,122-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:37037] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:37037]
-2013-11-06 12:22:01,122-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:37037]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fed
-2013-11-06 12:22:01,150-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fed
-MetaChannel [context: cps-37037, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:37037]
-2013-11-06 12:22:01,150-0600 INFO MetaChannel MetaChannel [context: cps-37037, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:37037]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 12:22:01,150-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 12:22:01,191-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 12:22:01,191-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 12:22:01,271-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 12:22:01,286-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 9.0) submitting urn:job-0-1383759274510-1383759274511: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 12:22:01,286-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 9.0) submitting urn:job-0-1383759274510-1383759274511: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 12:22:01,287-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 12:22:01,287-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-06 12:22:01,293-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7350, JOBSTATUS) unregistering (send)
-2013-11-06 12:22:01,294-0600 INFO RequestHandler Handler(tag: 7350, JOBSTATUS) unregistering (send)
-2013-11-06 12:22:03,319-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 12:22:03,319-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 12:22:03,319-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-06 12:22:03,319-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7351, JOBSTATUS) unregistering (send)
-2013-11-06 12:22:03,319-0600 INFO RequestHandler Handler(tag: 7351, JOBSTATUS) unregistering (send)
-2013-11-06 12:22:03,319-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 12:22:03,323-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:37038] for addition
-2013-11-06 12:22:03,324-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:37038] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:37038]
-2013-11-06 12:22:03,324-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:37038]
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7fec
-2013-11-06 12:22:03,358-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7fec
-MetaChannel [context: cps-37038, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:37038]
-2013-11-06 12:22:03,358-0600 INFO MetaChannel MetaChannel [context: cps-37038, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:37038]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-06 12:22:03,358-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-06 12:22:03,399-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 12:22:03,479-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 12:22:03,520-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 10.0) submitting urn:job-1-1383759274513-1383759274514: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 12:22:03,520-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 10.0) submitting urn:job-1-1383759274513-1383759274514: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 12:22:03,521-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 12:22:03,522-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-06 12:22:03,527-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7352, JOBSTATUS) unregistering (send)
-2013-11-06 12:22:03,527-0600 INFO RequestHandler Handler(tag: 7352, JOBSTATUS) unregistering (send)
-2013-11-06 12:22:05,622-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 12:22:05,622-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 12:22:05,622-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Handler(tag: 7353, JOBSTATUS) unregistering (send)
-2013-11-06 12:22:05,623-0600 INFO RequestHandler Handler(tag: 7353, JOBSTATUS) unregistering (send)
-2013-11-06 12:22:05,623-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-2013-11-06 12:22:05,623-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 12:22:05,624-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:37039] for addition
-2013-11-06 12:22:05,624-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:37039] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:37039]
-2013-11-06 12:22:05,624-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:37039]
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7feb
-2013-11-06 12:22:05,646-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7feb
-MetaChannel [context: cps-37039, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:37039]
-2013-11-06 12:22:05,646-0600 INFO MetaChannel MetaChannel [context: cps-37039, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:37039]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-06 12:22:05,646-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-06 12:22:05,687-0600 INFO RequestHandler Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-06 12:22:05,767-0600 INFO RequestHandler Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-06 12:22:05,773-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 11.0) submitting urn:job-2-1383759274516-1383759274517: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 12:22:05,773-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 11.0) submitting urn:job-2-1383759274516-1383759274517: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 12:22:05,774-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 12:22:05,774-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-06 12:22:05,779-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7354, JOBSTATUS) unregistering (send)
-2013-11-06 12:22:05,779-0600 INFO RequestHandler Handler(tag: 7354, JOBSTATUS) unregistering (send)
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 12:22:06,223-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 12:22:06,810-0600 INFO Multiplexer Avg stream buf: 0
-2013-11-06 12:22:07,794-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 12:22:07,794-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 12:22:07,794-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Handler(tag: 7355, JOBSTATUS) unregistering (send)
-2013-11-06 12:22:07,795-0600 INFO RequestHandler Handler(tag: 7355, JOBSTATUS) unregistering (send)
-2013-11-06 12:22:07,795-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-2013-11-06 12:22:07,795-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-Avg stream buf: 0
-2013-11-06 12:22:16,821-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 12:22:26,830-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:22:32,641-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 12:22:36,840-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:22:43,541-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 12:22:46,850-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 12:22:56,859-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 12:23:06,869-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:23:09,115-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 12:23:16,878-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 12:23:20,859-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 12:23:26,887-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 12:23:36,896-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:23:45,589-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 12:23:46,906-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 12:23:56,915-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 12:23:58,177-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122201.271, now=131106-122402.122, channel=TCPChannel [type: server, contact: 127.0.0.1:37037]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:24:02,122-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122201.271, now=131106-122402.122, channel=TCPChannel [type: server, contact: 127.0.0.1:37037]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 12:24:02,123-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122201.271, now=131106-122402.122, channel=TCPChannel [type: server, contact: 127.0.0.1:37037]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:24:02,123-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122201.271, now=131106-122402.122, channel=TCPChannel [type: server, contact: 127.0.0.1:37037]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 12:24:02,123-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122201.271, now=131106-122402.122, channel=TCPChannel [type: server, contact: 127.0.0.1:37037]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:24:02,123-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122201.271, now=131106-122402.122, channel=TCPChannel [type: server, contact: 127.0.0.1:37037]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 12:24:02,123-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122201.271, now=131106-122402.122, channel=TCPChannel [type: server, contact: 127.0.0.1:37037]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:24:02,123-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122201.271, now=131106-122402.122, channel=TCPChannel [type: server, contact: 127.0.0.1:37037]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 12:24:02,124-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122201.271, now=131106-122402.122, channel=TCPChannel [type: server, contact: 127.0.0.1:37037]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:24:02,124-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122201.271, now=131106-122402.122, channel=TCPChannel [type: server, contact: 127.0.0.1:37037]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 12:24:02,124-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:37037] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122201.271, now=131106-122402.122, channel=TCPChannel [type: server, contact: 127.0.0.1:37037]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:24:02,124-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:37037] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122201.271, now=131106-122402.122, channel=TCPChannel [type: server, contact: 127.0.0.1:37037]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-null: Channel shut down
-2013-11-06 12:24:02,124-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-37037, boundTo: TCPChannel [type: server, contact: 127.0.0.1:37037]] binding to NullChannel
-2013-11-06 12:24:02,124-0600 INFO MetaChannel MetaChannel [context: cps-37037, boundTo: TCPChannel [type: server, contact: 127.0.0.1:37037]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:37037]
-2013-11-06 12:24:02,124-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:37037]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122201.271, now=131106-122402.122, channel=TCPChannel [type: server, contact: 127.0.0.1:37037]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:24:02,124-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122201.271, now=131106-122402.122, channel=TCPChannel [type: server, contact: 127.0.0.1:37037]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 12:24:02,125-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122201.271, now=131106-122402.122, channel=TCPChannel [type: server, contact: 127.0.0.1:37037]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:24:02,125-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122201.271, now=131106-122402.122, channel=TCPChannel [type: server, contact: 127.0.0.1:37037]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 12:24:02,125-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122201.271, now=131106-122402.122, channel=TCPChannel [type: server, contact: 127.0.0.1:37037]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:24:02,125-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122201.271, now=131106-122402.122, channel=TCPChannel [type: server, contact: 127.0.0.1:37037]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 12:24:02,125-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122201.271, now=131106-122402.122, channel=TCPChannel [type: server, contact: 127.0.0.1:37037]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:24:02,125-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122201.271, now=131106-122402.122, channel=TCPChannel [type: server, contact: 127.0.0.1:37037]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 12:24:02,125-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122201.271, now=131106-122402.122, channel=TCPChannel [type: server, contact: 127.0.0.1:37037]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:24:02,125-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122201.271, now=131106-122402.122, channel=TCPChannel [type: server, contact: 127.0.0.1:37037]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel exception handled
-2013-11-06 12:24:02,126-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 12:24:02,130-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 12:24:02,130-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 12:24:02,131-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:37037] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 12:24:02,131-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:37037] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 12:24:02,131-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:37037]
-2013-11-06 12:24:02,131-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:37037]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122203.479, now=131106-122404.323, channel=TCPChannel [type: server, contact: 127.0.0.1:37038]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:24:04,323-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122203.479, now=131106-122404.323, channel=TCPChannel [type: server, contact: 127.0.0.1:37038]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 12:24:04,324-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122203.479, now=131106-122404.323, channel=TCPChannel [type: server, contact: 127.0.0.1:37038]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:24:04,324-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122203.479, now=131106-122404.323, channel=TCPChannel [type: server, contact: 127.0.0.1:37038]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 12:24:04,324-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122203.479, now=131106-122404.323, channel=TCPChannel [type: server, contact: 127.0.0.1:37038]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:24:04,324-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122203.479, now=131106-122404.323, channel=TCPChannel [type: server, contact: 127.0.0.1:37038]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 12:24:04,324-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122203.479, now=131106-122404.323, channel=TCPChannel [type: server, contact: 127.0.0.1:37038]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:24:04,324-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122203.479, now=131106-122404.323, channel=TCPChannel [type: server, contact: 127.0.0.1:37038]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 12:24:04,325-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122203.479, now=131106-122404.323, channel=TCPChannel [type: server, contact: 127.0.0.1:37038]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:24:04,325-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122203.479, now=131106-122404.323, channel=TCPChannel [type: server, contact: 127.0.0.1:37038]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 12:24:04,325-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:37038] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122203.479, now=131106-122404.323, channel=TCPChannel [type: server, contact: 127.0.0.1:37038]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:24:04,325-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:37038] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122203.479, now=131106-122404.323, channel=TCPChannel [type: server, contact: 127.0.0.1:37038]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-null: Channel shut down
-2013-11-06 12:24:04,325-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-37038, boundTo: TCPChannel [type: server, contact: 127.0.0.1:37038]] binding to NullChannel
-2013-11-06 12:24:04,325-0600 INFO MetaChannel MetaChannel [context: cps-37038, boundTo: TCPChannel [type: server, contact: 127.0.0.1:37038]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:37038]
-2013-11-06 12:24:04,326-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:37038]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122203.479, now=131106-122404.323, channel=TCPChannel [type: server, contact: 127.0.0.1:37038]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:24:04,326-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122203.479, now=131106-122404.323, channel=TCPChannel [type: server, contact: 127.0.0.1:37038]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 12:24:04,326-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122203.479, now=131106-122404.323, channel=TCPChannel [type: server, contact: 127.0.0.1:37038]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:24:04,326-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122203.479, now=131106-122404.323, channel=TCPChannel [type: server, contact: 127.0.0.1:37038]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 12:24:04,326-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122203.479, now=131106-122404.323, channel=TCPChannel [type: server, contact: 127.0.0.1:37038]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:24:04,326-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122203.479, now=131106-122404.323, channel=TCPChannel [type: server, contact: 127.0.0.1:37038]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 12:24:04,326-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122203.479, now=131106-122404.323, channel=TCPChannel [type: server, contact: 127.0.0.1:37038]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:24:04,326-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122203.479, now=131106-122404.323, channel=TCPChannel [type: server, contact: 127.0.0.1:37038]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 12:24:04,327-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122203.479, now=131106-122404.323, channel=TCPChannel [type: server, contact: 127.0.0.1:37038]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:24:04,327-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122203.479, now=131106-122404.323, channel=TCPChannel [type: server, contact: 127.0.0.1:37038]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel exception handled
-2013-11-06 12:24:04,327-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 12:24:04,338-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 12:24:04,338-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 12:24:04,338-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:37038] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 12:24:04,338-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:37038] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 12:24:04,338-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:37038]
-2013-11-06 12:24:04,338-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:37038]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122205.767, now=131106-122406.624, channel=TCPChannel [type: server, contact: 127.0.0.1:37039]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:24:06,624-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122205.767, now=131106-122406.624, channel=TCPChannel [type: server, contact: 127.0.0.1:37039]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 12:24:06,625-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122205.767, now=131106-122406.624, channel=TCPChannel [type: server, contact: 127.0.0.1:37039]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:24:06,625-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122205.767, now=131106-122406.624, channel=TCPChannel [type: server, contact: 127.0.0.1:37039]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 12:24:06,625-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122205.767, now=131106-122406.624, channel=TCPChannel [type: server, contact: 127.0.0.1:37039]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:24:06,625-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122205.767, now=131106-122406.624, channel=TCPChannel [type: server, contact: 127.0.0.1:37039]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 12:24:06,625-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122205.767, now=131106-122406.624, channel=TCPChannel [type: server, contact: 127.0.0.1:37039]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:24:06,625-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122205.767, now=131106-122406.624, channel=TCPChannel [type: server, contact: 127.0.0.1:37039]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 12:24:06,625-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122205.767, now=131106-122406.624, channel=TCPChannel [type: server, contact: 127.0.0.1:37039]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:24:06,625-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122205.767, now=131106-122406.624, channel=TCPChannel [type: server, contact: 127.0.0.1:37039]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 12:24:06,626-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:37039] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122205.767, now=131106-122406.624, channel=TCPChannel [type: server, contact: 127.0.0.1:37039]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:24:06,626-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:37039] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122205.767, now=131106-122406.624, channel=TCPChannel [type: server, contact: 127.0.0.1:37039]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-null: Channel shut down
-2013-11-06 12:24:06,626-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-37039, boundTo: TCPChannel [type: server, contact: 127.0.0.1:37039]] binding to NullChannel
-2013-11-06 12:24:06,626-0600 INFO MetaChannel MetaChannel [context: cps-37039, boundTo: TCPChannel [type: server, contact: 127.0.0.1:37039]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:37039]
-2013-11-06 12:24:06,626-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:37039]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122205.767, now=131106-122406.624, channel=TCPChannel [type: server, contact: 127.0.0.1:37039]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:24:06,626-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122205.767, now=131106-122406.624, channel=TCPChannel [type: server, contact: 127.0.0.1:37039]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 12:24:06,626-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122205.767, now=131106-122406.624, channel=TCPChannel [type: server, contact: 127.0.0.1:37039]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:24:06,627-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122205.767, now=131106-122406.624, channel=TCPChannel [type: server, contact: 127.0.0.1:37039]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 12:24:06,627-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122205.767, now=131106-122406.624, channel=TCPChannel [type: server, contact: 127.0.0.1:37039]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:24:06,627-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122205.767, now=131106-122406.624, channel=TCPChannel [type: server, contact: 127.0.0.1:37039]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 12:24:06,627-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122205.767, now=131106-122406.624, channel=TCPChannel [type: server, contact: 127.0.0.1:37039]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:24:06,627-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122205.767, now=131106-122406.624, channel=TCPChannel [type: server, contact: 127.0.0.1:37039]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 12:24:06,627-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122205.767, now=131106-122406.624, channel=TCPChannel [type: server, contact: 127.0.0.1:37039]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:24:06,627-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122205.767, now=131106-122406.624, channel=TCPChannel [type: server, contact: 127.0.0.1:37039]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel exception handled
-2013-11-06 12:24:06,628-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 12:24:06,645-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 12:24:06,645-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 12:24:06,645-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:37039] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 12:24:06,645-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:37039] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 12:24:06,645-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:37039]
-2013-11-06 12:24:06,645-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:37039]
-No streams
-2013-11-06 12:24:06,926-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:24:16,935-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:24:22,063-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:24:26,943-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:24:35,495-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:24:36,951-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:24:46,960-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:24:56,968-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:24:58,537-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:25:06,976-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:25:12,813-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:25:16,984-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:25:26,993-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 12:25:35,012-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 12:25:37,001-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:25:47,009-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:25:50,131-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:25:57,017-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:26:07,025-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:26:11,485-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:26:17,033-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:26:27,041-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 12:26:27,449-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 12:26:37,050-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:26:47,058-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:26:47,959-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:26:57,066-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 12:27:04,768-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 12:27:07,074-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:27:17,082-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:27:24,433-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:27:27,090-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:27:37,099-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:27:42,086-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:27:47,107-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:27:57,115-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:28:00,907-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 12:28:01,526-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:37046] for addition
-2013-11-06 12:28:01,526-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:37046] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:37046]
-2013-11-06 12:28:01,526-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:37046]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fea
-2013-11-06 12:28:01,548-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fea
-MetaChannel [context: cps-37046, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:37046]
-2013-11-06 12:28:01,549-0600 INFO MetaChannel MetaChannel [context: cps-37046, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:37046]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 12:28:01,549-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 12:28:01,589-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 12:28:01,589-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 12:28:01,670-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 12:28:01,688-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 12.0) submitting urn:job-0-1383759274519-1383759274520: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 12:28:01,689-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 12.0) submitting urn:job-0-1383759274519-1383759274520: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 12:28:01,689-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 12:28:01,689-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-06 12:28:01,695-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7356, JOBSTATUS) unregistering (send)
-2013-11-06 12:28:01,695-0600 INFO RequestHandler Handler(tag: 7356, JOBSTATUS) unregistering (send)
-2013-11-06 12:28:03,833-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 12:28:03,833-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 12:28:03,833-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-06 12:28:03,833-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7357, JOBSTATUS) unregistering (send)
-2013-11-06 12:28:03,833-0600 INFO RequestHandler Handler(tag: 7357, JOBSTATUS) unregistering (send)
-2013-11-06 12:28:03,833-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-Avg stream buf: 0
-2013-11-06 12:28:07,126-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 12:28:17,134-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 12:28:19,404-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 12:28:24,296-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:37047] for addition
-2013-11-06 12:28:24,297-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:37047] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:37047]
-2013-11-06 12:28:24,297-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:37047]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fe9
-2013-11-06 12:28:24,336-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fe9
-MetaChannel [context: cps-37047, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:37047]
-2013-11-06 12:28:24,336-0600 INFO MetaChannel MetaChannel [context: cps-37047, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:37047]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 12:28:24,336-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 12:28:24,357-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 12:28:24,357-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 12:28:24,437-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 12:28:24,457-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 13.0) submitting urn:job-0-1383759274522-1383759274523: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 12:28:24,457-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 13.0) submitting urn:job-0-1383759274522-1383759274523: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 12:28:24,457-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 12:28:24,458-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-06 12:28:24,463-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7358, JOBSTATUS) unregistering (send)
-2013-11-06 12:28:24,464-0600 INFO RequestHandler Handler(tag: 7358, JOBSTATUS) unregistering (send)
-2013-11-06 12:28:26,481-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 12:28:26,481-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 12:28:26,481-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-06 12:28:26,481-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7359, JOBSTATUS) unregistering (send)
-2013-11-06 12:28:26,481-0600 INFO RequestHandler Handler(tag: 7359, JOBSTATUS) unregistering (send)
-2013-11-06 12:28:26,481-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 12:28:26,483-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:37048] for addition
-2013-11-06 12:28:26,483-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:37048] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:37048]
-2013-11-06 12:28:26,483-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:37048]
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7fe8
-2013-11-06 12:28:26,504-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7fe8
-MetaChannel [context: cps-37048, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:37048]
-2013-11-06 12:28:26,504-0600 INFO MetaChannel MetaChannel [context: cps-37048, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:37048]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-06 12:28:26,504-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-06 12:28:26,544-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 12:28:26,625-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 12:28:26,632-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 14.0) submitting urn:job-1-1383759274525-1383759274526: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 12:28:26,632-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 14.0) submitting urn:job-1-1383759274525-1383759274526: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 12:28:26,632-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 12:28:26,632-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-06 12:28:26,638-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7360, JOBSTATUS) unregistering (send)
-2013-11-06 12:28:26,638-0600 INFO RequestHandler Handler(tag: 7360, JOBSTATUS) unregistering (send)
-Avg stream buf: 0
-2013-11-06 12:28:27,146-0600 INFO Multiplexer Avg stream buf: 0
-2013-11-06 12:28:28,652-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 12:28:28,652-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 12:28:28,652-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-06 12:28:28,652-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7361, JOBSTATUS) unregistering (send)
-2013-11-06 12:28:28,653-0600 INFO RequestHandler Handler(tag: 7361, JOBSTATUS) unregistering (send)
-2013-11-06 12:28:28,653-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 12:28:28,653-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:37049] for addition
-2013-11-06 12:28:28,654-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:37049] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:37049]
-2013-11-06 12:28:28,654-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:37049]
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7fe7
-2013-11-06 12:28:28,689-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7fe7
-MetaChannel [context: cps-37049, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:37049]
-2013-11-06 12:28:28,689-0600 INFO MetaChannel MetaChannel [context: cps-37049, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:37049]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-06 12:28:28,689-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-Avg stream buf: 0
-2013-11-06 12:28:37,154-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:28:37,381-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 12:28:47,164-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:28:56,722-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 12:28:57,173-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 12:29:07,182-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:29:13,855-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 12:29:17,191-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 12:29:27,201-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:29:34,040-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 12:29:37,210-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 12:29:47,219-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:29:50,329-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 12:29:57,229-0600 INFO Multiplexer Avg stream buf: 0
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122801.669, now=131106-123002.526, channel=TCPChannel [type: server, contact: 127.0.0.1:37046]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:30:02,526-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122801.669, now=131106-123002.526, channel=TCPChannel [type: server, contact: 127.0.0.1:37046]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 12:30:02,526-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122801.669, now=131106-123002.526, channel=TCPChannel [type: server, contact: 127.0.0.1:37046]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:30:02,526-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122801.669, now=131106-123002.526, channel=TCPChannel [type: server, contact: 127.0.0.1:37046]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 12:30:02,527-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122801.669, now=131106-123002.526, channel=TCPChannel [type: server, contact: 127.0.0.1:37046]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:30:02,527-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122801.669, now=131106-123002.526, channel=TCPChannel [type: server, contact: 127.0.0.1:37046]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 12:30:02,527-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122801.669, now=131106-123002.526, channel=TCPChannel [type: server, contact: 127.0.0.1:37046]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:30:02,527-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122801.669, now=131106-123002.526, channel=TCPChannel [type: server, contact: 127.0.0.1:37046]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 12:30:02,527-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122801.669, now=131106-123002.526, channel=TCPChannel [type: server, contact: 127.0.0.1:37046]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:30:02,527-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122801.669, now=131106-123002.526, channel=TCPChannel [type: server, contact: 127.0.0.1:37046]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 12:30:02,527-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:37046] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122801.669, now=131106-123002.526, channel=TCPChannel [type: server, contact: 127.0.0.1:37046]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:30:02,528-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:37046] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122801.669, now=131106-123002.526, channel=TCPChannel [type: server, contact: 127.0.0.1:37046]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-null: Channel shut down
-2013-11-06 12:30:02,528-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-37046, boundTo: TCPChannel [type: server, contact: 127.0.0.1:37046]] binding to NullChannel
-2013-11-06 12:30:02,528-0600 INFO MetaChannel MetaChannel [context: cps-37046, boundTo: TCPChannel [type: server, contact: 127.0.0.1:37046]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:37046]
-2013-11-06 12:30:02,528-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:37046]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122801.669, now=131106-123002.526, channel=TCPChannel [type: server, contact: 127.0.0.1:37046]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:30:02,528-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122801.669, now=131106-123002.526, channel=TCPChannel [type: server, contact: 127.0.0.1:37046]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 12:30:02,528-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122801.669, now=131106-123002.526, channel=TCPChannel [type: server, contact: 127.0.0.1:37046]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:30:02,528-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122801.669, now=131106-123002.526, channel=TCPChannel [type: server, contact: 127.0.0.1:37046]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 12:30:02,529-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122801.669, now=131106-123002.526, channel=TCPChannel [type: server, contact: 127.0.0.1:37046]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:30:02,529-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122801.669, now=131106-123002.526, channel=TCPChannel [type: server, contact: 127.0.0.1:37046]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 12:30:02,529-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122801.669, now=131106-123002.526, channel=TCPChannel [type: server, contact: 127.0.0.1:37046]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:30:02,529-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122801.669, now=131106-123002.526, channel=TCPChannel [type: server, contact: 127.0.0.1:37046]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 12:30:02,529-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122801.669, now=131106-123002.526, channel=TCPChannel [type: server, contact: 127.0.0.1:37046]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:30:02,529-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122801.669, now=131106-123002.526, channel=TCPChannel [type: server, contact: 127.0.0.1:37046]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel exception handled
-2013-11-06 12:30:02,529-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 12:30:02,544-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 12:30:02,544-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 12:30:02,544-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:37046] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 12:30:02,545-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:37046] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 12:30:02,545-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:37046]
-2013-11-06 12:30:02,545-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:37046]
-Avg stream buf: 0
-2013-11-06 12:30:07,238-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:30:11,358-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 12:30:17,247-0600 INFO Multiplexer Avg stream buf: 0
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122824.437, now=131106-123025.296, channel=TCPChannel [type: server, contact: 127.0.0.1:37047]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:30:25,297-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122824.437, now=131106-123025.296, channel=TCPChannel [type: server, contact: 127.0.0.1:37047]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 12:30:25,297-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122824.437, now=131106-123025.296, channel=TCPChannel [type: server, contact: 127.0.0.1:37047]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:30:25,297-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122824.437, now=131106-123025.296, channel=TCPChannel [type: server, contact: 127.0.0.1:37047]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 12:30:25,297-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122824.437, now=131106-123025.296, channel=TCPChannel [type: server, contact: 127.0.0.1:37047]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:30:25,297-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122824.437, now=131106-123025.296, channel=TCPChannel [type: server, contact: 127.0.0.1:37047]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 12:30:25,298-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122824.437, now=131106-123025.296, channel=TCPChannel [type: server, contact: 127.0.0.1:37047]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:30:25,298-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122824.437, now=131106-123025.296, channel=TCPChannel [type: server, contact: 127.0.0.1:37047]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 12:30:25,298-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122824.437, now=131106-123025.296, channel=TCPChannel [type: server, contact: 127.0.0.1:37047]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:30:25,298-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122824.437, now=131106-123025.296, channel=TCPChannel [type: server, contact: 127.0.0.1:37047]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 12:30:25,298-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:37047] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122824.437, now=131106-123025.296, channel=TCPChannel [type: server, contact: 127.0.0.1:37047]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:30:25,298-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:37047] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122824.437, now=131106-123025.296, channel=TCPChannel [type: server, contact: 127.0.0.1:37047]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-null: Channel shut down
-2013-11-06 12:30:25,299-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-37047, boundTo: TCPChannel [type: server, contact: 127.0.0.1:37047]] binding to NullChannel
-2013-11-06 12:30:25,299-0600 INFO MetaChannel MetaChannel [context: cps-37047, boundTo: TCPChannel [type: server, contact: 127.0.0.1:37047]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:37047]
-2013-11-06 12:30:25,299-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:37047]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122824.437, now=131106-123025.296, channel=TCPChannel [type: server, contact: 127.0.0.1:37047]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:30:25,299-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122824.437, now=131106-123025.296, channel=TCPChannel [type: server, contact: 127.0.0.1:37047]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 12:30:25,299-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122824.437, now=131106-123025.296, channel=TCPChannel [type: server, contact: 127.0.0.1:37047]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:30:25,299-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122824.437, now=131106-123025.296, channel=TCPChannel [type: server, contact: 127.0.0.1:37047]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 12:30:25,299-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122824.437, now=131106-123025.296, channel=TCPChannel [type: server, contact: 127.0.0.1:37047]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:30:25,299-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122824.437, now=131106-123025.296, channel=TCPChannel [type: server, contact: 127.0.0.1:37047]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 12:30:25,300-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122824.437, now=131106-123025.296, channel=TCPChannel [type: server, contact: 127.0.0.1:37047]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:30:25,300-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122824.437, now=131106-123025.296, channel=TCPChannel [type: server, contact: 127.0.0.1:37047]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 12:30:25,300-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122824.437, now=131106-123025.296, channel=TCPChannel [type: server, contact: 127.0.0.1:37047]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:30:25,300-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122824.437, now=131106-123025.296, channel=TCPChannel [type: server, contact: 127.0.0.1:37047]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel exception handled
-2013-11-06 12:30:25,300-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 12:30:25,311-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 12:30:25,311-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 12:30:25,311-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:37047] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 12:30:25,311-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:37047] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 12:30:25,312-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:37047]
-2013-11-06 12:30:25,312-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:37047]
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:30:26,803-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 12:30:27,257-0600 INFO Multiplexer Avg stream buf: 0
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122826.625, now=131106-123027.483, channel=TCPChannel [type: server, contact: 127.0.0.1:37048]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:30:27,483-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122826.625, now=131106-123027.483, channel=TCPChannel [type: server, contact: 127.0.0.1:37048]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 12:30:27,483-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122826.625, now=131106-123027.483, channel=TCPChannel [type: server, contact: 127.0.0.1:37048]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:30:27,483-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122826.625, now=131106-123027.483, channel=TCPChannel [type: server, contact: 127.0.0.1:37048]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 12:30:27,484-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122826.625, now=131106-123027.483, channel=TCPChannel [type: server, contact: 127.0.0.1:37048]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:30:27,484-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122826.625, now=131106-123027.483, channel=TCPChannel [type: server, contact: 127.0.0.1:37048]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 12:30:27,484-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122826.625, now=131106-123027.483, channel=TCPChannel [type: server, contact: 127.0.0.1:37048]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:30:27,484-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122826.625, now=131106-123027.483, channel=TCPChannel [type: server, contact: 127.0.0.1:37048]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 12:30:27,484-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122826.625, now=131106-123027.483, channel=TCPChannel [type: server, contact: 127.0.0.1:37048]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:30:27,484-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122826.625, now=131106-123027.483, channel=TCPChannel [type: server, contact: 127.0.0.1:37048]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 12:30:27,485-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:37048] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122826.625, now=131106-123027.483, channel=TCPChannel [type: server, contact: 127.0.0.1:37048]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:30:27,485-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:37048] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122826.625, now=131106-123027.483, channel=TCPChannel [type: server, contact: 127.0.0.1:37048]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-null: Channel shut down
-2013-11-06 12:30:27,485-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-37048, boundTo: TCPChannel [type: server, contact: 127.0.0.1:37048]] binding to NullChannel
-2013-11-06 12:30:27,485-0600 INFO MetaChannel MetaChannel [context: cps-37048, boundTo: TCPChannel [type: server, contact: 127.0.0.1:37048]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:37048]
-2013-11-06 12:30:27,485-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:37048]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122826.625, now=131106-123027.483, channel=TCPChannel [type: server, contact: 127.0.0.1:37048]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:30:27,485-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122826.625, now=131106-123027.483, channel=TCPChannel [type: server, contact: 127.0.0.1:37048]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 12:30:27,486-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122826.625, now=131106-123027.483, channel=TCPChannel [type: server, contact: 127.0.0.1:37048]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:30:27,486-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122826.625, now=131106-123027.483, channel=TCPChannel [type: server, contact: 127.0.0.1:37048]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 12:30:27,486-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122826.625, now=131106-123027.483, channel=TCPChannel [type: server, contact: 127.0.0.1:37048]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:30:27,486-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122826.625, now=131106-123027.483, channel=TCPChannel [type: server, contact: 127.0.0.1:37048]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 12:30:27,486-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122826.625, now=131106-123027.483, channel=TCPChannel [type: server, contact: 127.0.0.1:37048]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:30:27,486-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122826.625, now=131106-123027.483, channel=TCPChannel [type: server, contact: 127.0.0.1:37048]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 12:30:27,486-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122826.625, now=131106-123027.483, channel=TCPChannel [type: server, contact: 127.0.0.1:37048]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:30:27,486-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122826.625, now=131106-123027.483, channel=TCPChannel [type: server, contact: 127.0.0.1:37048]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel exception handled
-2013-11-06 12:30:27,487-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 12:30:27,498-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 12:30:27,498-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 12:30:27,498-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:37048] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 12:30:27,498-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:37048] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 12:30:27,499-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:37048]
-2013-11-06 12:30:27,499-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:37048]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122828.689, now=131106-123029.653, channel=TCPChannel [type: server, contact: 127.0.0.1:37049]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:30:29,653-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122828.689, now=131106-123029.653, channel=TCPChannel [type: server, contact: 127.0.0.1:37049]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 12:30:29,654-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:37049] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122828.689, now=131106-123029.653, channel=TCPChannel [type: server, contact: 127.0.0.1:37049]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:30:29,654-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:37049] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122828.689, now=131106-123029.653, channel=TCPChannel [type: server, contact: 127.0.0.1:37049]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-null: Channel shut down
-2013-11-06 12:30:29,654-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-37049, boundTo: TCPChannel [type: server, contact: 127.0.0.1:37049]] binding to NullChannel
-2013-11-06 12:30:29,654-0600 INFO MetaChannel MetaChannel [context: cps-37049, boundTo: TCPChannel [type: server, contact: 127.0.0.1:37049]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:37049]
-2013-11-06 12:30:29,654-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:37049]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122828.689, now=131106-123029.653, channel=TCPChannel [type: server, contact: 127.0.0.1:37049]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:30:29,654-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-122828.689, now=131106-123029.653, channel=TCPChannel [type: server, contact: 127.0.0.1:37049]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel exception handled
-2013-11-06 12:30:29,655-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 12:30:29,665-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 12:30:29,665-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 12:30:29,665-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:37049] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 12:30:29,665-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:37049] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 12:30:29,666-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:37049]
-2013-11-06 12:30:29,666-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:37049]
-No streams
-2013-11-06 12:30:37,267-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:30:47,275-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:30:48,676-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:30:57,283-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 12:31:03,278-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 12:31:07,291-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:31:17,299-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:31:25,994-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:31:27,307-0600 INFO Multiplexer No streams
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 12:31:33,905-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:37051] for addition
-2013-11-06 12:31:33,905-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:37051] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:37051]
-2013-11-06 12:31:33,905-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:37051]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fe6
-2013-11-06 12:31:33,927-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fe6
-MetaChannel [context: cps-37051, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:37051]
-2013-11-06 12:31:33,927-0600 INFO MetaChannel MetaChannel [context: cps-37051, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:37051]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 12:31:33,927-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 12:31:33,968-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 12:31:33,968-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 12:31:34,048-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 12:31:34,063-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 15.0) submitting urn:job-0-1383759274528-1383759274529: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 12:31:34,063-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 15.0) submitting urn:job-0-1383759274528-1383759274529: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 12:31:34,064-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 12:31:34,064-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-06 12:31:34,070-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7362, JOBSTATUS) unregistering (send)
-2013-11-06 12:31:34,070-0600 INFO RequestHandler Handler(tag: 7362, JOBSTATUS) unregistering (send)
-2013-11-06 12:31:36,108-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 12:31:36,108-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 12:31:36,108-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-06 12:31:36,108-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7363, JOBSTATUS) unregistering (send)
-2013-11-06 12:31:36,108-0600 INFO RequestHandler Handler(tag: 7363, JOBSTATUS) unregistering (send)
-2013-11-06 12:31:36,108-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 12:31:36,110-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:37052] for addition
-2013-11-06 12:31:36,110-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:37052] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:37052]
-2013-11-06 12:31:36,110-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:37052]
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7fe5
-2013-11-06 12:31:36,135-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7fe5
-MetaChannel [context: cps-37052, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:37052]
-2013-11-06 12:31:36,135-0600 INFO MetaChannel MetaChannel [context: cps-37052, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:37052]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-06 12:31:36,135-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-06 12:31:36,175-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 12:31:36,256-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 12:31:36,259-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 16.0) submitting urn:job-1-1383759274531-1383759274532: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 12:31:36,259-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 16.0) submitting urn:job-1-1383759274531-1383759274532: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 12:31:36,259-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 12:31:36,259-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-06 12:31:36,267-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7364, JOBSTATUS) unregistering (send)
-2013-11-06 12:31:36,268-0600 INFO RequestHandler Handler(tag: 7364, JOBSTATUS) unregistering (send)
-Avg stream buf: 0
-2013-11-06 12:31:37,319-0600 INFO Multiplexer Avg stream buf: 0
-2013-11-06 12:31:38,279-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 12:31:38,280-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 12:31:38,280-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-06 12:31:38,280-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7365, JOBSTATUS) unregistering (send)
-2013-11-06 12:31:38,280-0600 INFO RequestHandler Handler(tag: 7365, JOBSTATUS) unregistering (send)
-2013-11-06 12:31:38,280-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 12:31:38,281-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:37053] for addition
-2013-11-06 12:31:38,282-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:37053] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:37053]
-2013-11-06 12:31:38,282-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:37053]
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7fe4
-2013-11-06 12:31:38,304-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7fe4
-MetaChannel [context: cps-37053, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:37053]
-2013-11-06 12:31:38,304-0600 INFO MetaChannel MetaChannel [context: cps-37053, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:37053]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-06 12:31:38,305-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:31:39,751-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 12:31:47,331-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 12:31:57,340-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:32:03,312-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 12:32:07,349-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 12:32:16,226-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-Avg stream buf: 0
-2013-11-06 12:32:17,358-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 12:32:27,368-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 12:32:37,377-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 12:32:40,630-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-Avg stream buf: 0
-2013-11-06 12:32:47,386-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:32:52,699-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 12:32:57,395-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 12:33:07,404-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 12:33:17,413-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 12:33:17,949-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 12:33:27,422-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:33:29,173-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-123134.048, now=131106-123334.905, channel=TCPChannel [type: server, contact: 127.0.0.1:37051]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:33:34,905-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-123134.048, now=131106-123334.905, channel=TCPChannel [type: server, contact: 127.0.0.1:37051]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 12:33:34,905-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-123134.048, now=131106-123334.905, channel=TCPChannel [type: server, contact: 127.0.0.1:37051]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:33:34,906-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-123134.048, now=131106-123334.905, channel=TCPChannel [type: server, contact: 127.0.0.1:37051]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 12:33:34,906-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-123134.048, now=131106-123334.905, channel=TCPChannel [type: server, contact: 127.0.0.1:37051]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:33:34,906-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-123134.048, now=131106-123334.905, channel=TCPChannel [type: server, contact: 127.0.0.1:37051]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 12:33:34,906-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-123134.048, now=131106-123334.905, channel=TCPChannel [type: server, contact: 127.0.0.1:37051]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:33:34,906-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-123134.048, now=131106-123334.905, channel=TCPChannel [type: server, contact: 127.0.0.1:37051]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 12:33:34,907-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-123134.048, now=131106-123334.905, channel=TCPChannel [type: server, contact: 127.0.0.1:37051]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:33:34,907-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-123134.048, now=131106-123334.905, channel=TCPChannel [type: server, contact: 127.0.0.1:37051]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 12:33:34,907-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:37051] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-123134.048, now=131106-123334.905, channel=TCPChannel [type: server, contact: 127.0.0.1:37051]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:33:34,907-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:37051] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-123134.048, now=131106-123334.905, channel=TCPChannel [type: server, contact: 127.0.0.1:37051]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-null: Channel shut down
-2013-11-06 12:33:34,907-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-37051, boundTo: TCPChannel [type: server, contact: 127.0.0.1:37051]] binding to NullChannel
-2013-11-06 12:33:34,907-0600 INFO MetaChannel MetaChannel [context: cps-37051, boundTo: TCPChannel [type: server, contact: 127.0.0.1:37051]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:37051]
-2013-11-06 12:33:34,907-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:37051]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-123134.048, now=131106-123334.905, channel=TCPChannel [type: server, contact: 127.0.0.1:37051]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:33:34,908-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-123134.048, now=131106-123334.905, channel=TCPChannel [type: server, contact: 127.0.0.1:37051]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 12:33:34,908-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-123134.048, now=131106-123334.905, channel=TCPChannel [type: server, contact: 127.0.0.1:37051]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:33:34,908-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-123134.048, now=131106-123334.905, channel=TCPChannel [type: server, contact: 127.0.0.1:37051]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 12:33:34,908-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-123134.048, now=131106-123334.905, channel=TCPChannel [type: server, contact: 127.0.0.1:37051]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:33:34,908-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-123134.048, now=131106-123334.905, channel=TCPChannel [type: server, contact: 127.0.0.1:37051]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 12:33:34,908-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-123134.048, now=131106-123334.905, channel=TCPChannel [type: server, contact: 127.0.0.1:37051]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:33:34,908-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-123134.048, now=131106-123334.905, channel=TCPChannel [type: server, contact: 127.0.0.1:37051]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 12:33:34,908-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-123134.048, now=131106-123334.905, channel=TCPChannel [type: server, contact: 127.0.0.1:37051]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:33:34,909-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-123134.048, now=131106-123334.905, channel=TCPChannel [type: server, contact: 127.0.0.1:37051]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel exception handled
-2013-11-06 12:33:34,909-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 12:33:34,924-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 12:33:34,924-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 12:33:34,924-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:37051] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 12:33:34,924-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:37051] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 12:33:34,925-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:37051]
-2013-11-06 12:33:34,925-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:37051]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-123136.256, now=131106-123337.110, channel=TCPChannel [type: server, contact: 127.0.0.1:37052]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:33:37,110-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-123136.256, now=131106-123337.110, channel=TCPChannel [type: server, contact: 127.0.0.1:37052]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 12:33:37,111-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-123136.256, now=131106-123337.110, channel=TCPChannel [type: server, contact: 127.0.0.1:37052]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:33:37,111-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-123136.256, now=131106-123337.110, channel=TCPChannel [type: server, contact: 127.0.0.1:37052]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 12:33:37,111-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-123136.256, now=131106-123337.110, channel=TCPChannel [type: server, contact: 127.0.0.1:37052]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:33:37,111-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-123136.256, now=131106-123337.110, channel=TCPChannel [type: server, contact: 127.0.0.1:37052]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 12:33:37,111-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-123136.256, now=131106-123337.110, channel=TCPChannel [type: server, contact: 127.0.0.1:37052]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:33:37,111-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-123136.256, now=131106-123337.110, channel=TCPChannel [type: server, contact: 127.0.0.1:37052]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 12:33:37,112-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-123136.256, now=131106-123337.110, channel=TCPChannel [type: server, contact: 127.0.0.1:37052]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:33:37,112-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-123136.256, now=131106-123337.110, channel=TCPChannel [type: server, contact: 127.0.0.1:37052]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 12:33:37,112-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:37052] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-123136.256, now=131106-123337.110, channel=TCPChannel [type: server, contact: 127.0.0.1:37052]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:33:37,112-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:37052] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-123136.256, now=131106-123337.110, channel=TCPChannel [type: server, contact: 127.0.0.1:37052]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-null: Channel shut down
-2013-11-06 12:33:37,112-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-37052, boundTo: TCPChannel [type: server, contact: 127.0.0.1:37052]] binding to NullChannel
-2013-11-06 12:33:37,113-0600 INFO MetaChannel MetaChannel [context: cps-37052, boundTo: TCPChannel [type: server, contact: 127.0.0.1:37052]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:37052]
-2013-11-06 12:33:37,113-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:37052]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-123136.256, now=131106-123337.110, channel=TCPChannel [type: server, contact: 127.0.0.1:37052]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:33:37,113-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-123136.256, now=131106-123337.110, channel=TCPChannel [type: server, contact: 127.0.0.1:37052]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 12:33:37,113-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-123136.256, now=131106-123337.110, channel=TCPChannel [type: server, contact: 127.0.0.1:37052]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:33:37,113-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-123136.256, now=131106-123337.110, channel=TCPChannel [type: server, contact: 127.0.0.1:37052]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 12:33:37,113-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-123136.256, now=131106-123337.110, channel=TCPChannel [type: server, contact: 127.0.0.1:37052]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:33:37,113-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-123136.256, now=131106-123337.110, channel=TCPChannel [type: server, contact: 127.0.0.1:37052]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 12:33:37,113-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-123136.256, now=131106-123337.110, channel=TCPChannel [type: server, contact: 127.0.0.1:37052]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:33:37,113-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-123136.256, now=131106-123337.110, channel=TCPChannel [type: server, contact: 127.0.0.1:37052]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 12:33:37,114-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-123136.256, now=131106-123337.110, channel=TCPChannel [type: server, contact: 127.0.0.1:37052]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:33:37,114-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-123136.256, now=131106-123337.110, channel=TCPChannel [type: server, contact: 127.0.0.1:37052]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel exception handled
-2013-11-06 12:33:37,114-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 12:33:37,131-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 12:33:37,131-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 12:33:37,131-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:37052] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 12:33:37,132-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:37052] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 12:33:37,132-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:37052]
-2013-11-06 12:33:37,132-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:37052]
-Avg stream buf: 0
-2013-11-06 12:33:37,433-0600 INFO Multiplexer Avg stream buf: 0
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-123138.304, now=131106-123339.282, channel=TCPChannel [type: server, contact: 127.0.0.1:37053]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:33:39,282-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-123138.304, now=131106-123339.282, channel=TCPChannel [type: server, contact: 127.0.0.1:37053]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 12:33:39,282-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:37053] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-123138.304, now=131106-123339.282, channel=TCPChannel [type: server, contact: 127.0.0.1:37053]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:33:39,282-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:37053] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-123138.304, now=131106-123339.282, channel=TCPChannel [type: server, contact: 127.0.0.1:37053]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-null: Channel shut down
-2013-11-06 12:33:39,282-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-37053, boundTo: TCPChannel [type: server, contact: 127.0.0.1:37053]] binding to NullChannel
-2013-11-06 12:33:39,282-0600 INFO MetaChannel MetaChannel [context: cps-37053, boundTo: TCPChannel [type: server, contact: 127.0.0.1:37053]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:37053]
-2013-11-06 12:33:39,283-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:37053]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-123138.304, now=131106-123339.282, channel=TCPChannel [type: server, contact: 127.0.0.1:37053]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 12:33:39,283-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-123138.304, now=131106-123339.282, channel=TCPChannel [type: server, contact: 127.0.0.1:37053]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel exception handled
-2013-11-06 12:33:39,283-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 12:33:39,298-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 12:33:39,298-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 12:33:39,298-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:37053] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 12:33:39,299-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:37053] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 12:33:39,299-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:37053]
-2013-11-06 12:33:39,299-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:37053]
-No streams
-2013-11-06 12:33:47,442-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 12:33:55,267-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 12:33:57,450-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:34:05,647-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:34:07,458-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:34:17,467-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:34:27,475-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 12:34:32,584-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 12:34:37,483-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 12:34:42,122-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 12:34:47,491-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:34:57,499-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:35:07,507-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 12:35:09,903-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 12:35:17,515-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:35:18,595-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:35:27,523-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:35:37,531-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 12:35:47,222-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 12:35:47,539-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:35:55,070-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:35:57,547-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:36:07,556-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:36:17,564-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:36:24,539-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:36:27,572-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:36:31,544-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:36:37,580-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:36:47,588-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:36:57,597-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 12:37:01,857-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 12:37:07,605-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:37:08,018-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:37:17,613-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:37:27,621-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:37:37,630-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 12:37:39,175-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:37:44,492-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:37:47,638-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:37:57,646-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:38:07,654-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:38:16,493-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:38:17,662-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:38:20,966-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:38:27,671-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:38:37,679-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:38:47,687-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 12:38:53,811-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:38:57,440-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:38:57,695-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:39:07,703-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:39:17,711-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:39:27,720-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 12:39:31,129-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:39:33,914-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:39:37,728-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:39:47,736-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:39:57,744-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:40:07,752-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 12:40:08,447-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:40:10,388-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:40:17,760-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:40:27,768-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:40:37,777-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:40:45,766-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:40:46,862-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:40:47,785-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:40:57,793-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:41:07,801-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:41:17,809-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:41:23,084-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 12:41:23,337-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 12:41:27,817-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:41:37,825-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:41:47,833-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:41:57,841-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:41:59,810-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 12:42:00,402-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 12:42:07,849-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:42:17,857-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:42:27,865-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:42:36,284-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 12:42:37,720-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 12:42:37,873-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:42:47,881-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:42:57,889-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:43:07,897-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:43:12,758-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:43:15,038-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:43:17,905-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:43:27,913-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:43:37,921-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:43:47,929-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 12:43:49,232-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:43:52,356-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:43:57,937-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:44:07,945-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:44:17,953-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:44:25,706-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:44:27,961-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 12:44:29,674-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 12:44:37,969-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:44:47,977-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:44:57,984-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:45:02,180-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 12:45:06,991-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 12:45:07,992-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:45:17,999-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:45:28,006-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:45:38,014-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 12:45:38,655-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:45:44,310-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:45:48,022-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:45:58,029-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:46:08,037-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:46:15,128-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:46:18,045-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 12:46:21,628-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 12:46:28,052-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:46:38,060-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:46:48,068-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:46:51,602-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:46:58,075-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 12:46:58,946-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 12:47:08,083-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:47:18,091-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:47:28,076-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:47:28,098-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 12:47:36,263-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 12:47:38,106-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:47:48,114-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:47:58,121-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 12:48:04,551-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 12:48:08,129-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:48:13,582-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:48:18,136-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:48:28,144-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:48:38,152-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 12:48:41,024-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 12:48:48,159-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 12:48:50,900-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 12:48:58,167-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:49:08,174-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:49:17,498-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:49:18,182-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:49:28,190-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 12:49:28,218-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 12:49:38,197-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:49:48,205-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:49:53,972-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:49:58,212-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:50:05,536-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:50:08,219-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:50:18,227-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:50:28,235-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:50:30,446-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:50:38,243-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:50:42,854-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:50:48,250-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:50:58,258-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:51:06,920-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:51:08,265-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:51:18,273-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 12:51:20,172-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 12:51:28,280-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:51:38,288-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:51:43,394-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:51:48,295-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 12:51:57,490-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 12:51:58,303-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:52:08,310-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:52:18,318-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 12:52:19,869-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 12:52:28,326-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:52:34,808-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:52:38,333-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:52:48,341-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:52:56,343-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:52:58,349-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:53:08,356-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:53:12,126-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:53:18,364-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:53:28,371-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 12:53:32,817-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 12:53:38,379-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:53:48,386-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 12:53:49,444-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 12:53:58,394-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:54:08,401-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:54:09,291-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:54:18,409-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 12:54:26,763-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 12:54:28,416-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:54:38,424-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 12:54:45,766-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 12:54:48,432-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:54:58,439-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:55:04,081-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:55:08,447-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:55:18,454-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 12:55:22,239-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 12:55:28,462-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:55:38,469-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:55:41,399-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:55:48,477-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:55:58,484-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:55:58,713-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:56:08,492-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:56:18,500-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 12:56:18,717-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 12:56:28,507-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:56:35,187-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:56:38,515-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:56:48,522-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 12:56:56,036-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 12:56:58,530-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:57:08,537-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:57:11,662-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:57:18,545-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:57:28,553-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:57:33,354-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:57:38,562-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 12:57:48,137-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 12:57:48,571-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:57:58,580-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:58:08,588-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:58:10,672-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:58:18,597-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 12:58:24,611-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 12:58:28,605-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:58:38,614-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 12:58:47,991-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 12:58:48,622-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:58:58,631-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 12:59:01,084-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 12:59:08,639-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:59:18,648-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 12:59:25,308-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 12:59:28,659-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 12:59:37,559-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 12:59:38,670-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:59:48,680-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 12:59:58,691-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:00:02,626-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:00:08,703-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 13:00:14,033-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 13:00:18,714-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:00:28,725-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:00:38,735-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 13:00:39,944-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 13:00:48,746-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:00:50,506-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:00:58,765-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:01:08,781-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 13:01:17,263-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 13:01:18,788-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 13:01:26,981-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 13:01:28,793-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:01:38,797-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:01:48,802-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 13:01:54,581-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 13:01:58,804-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 13:02:03,455-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 13:02:08,806-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:02:18,814-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:02:28,823-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:02:31,898-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:02:38,831-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 13:02:39,929-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 13:02:48,843-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:02:58,854-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:03:08,864-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:03:09,216-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 13:03:16,403-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 13:03:18,877-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:03:28,887-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:03:38,896-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 13:03:46,535-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 13:03:48,906-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:03:52,877-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:03:58,914-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:04:08,924-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:04:18,934-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:04:23,852-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:04:28,943-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 13:04:29,352-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 13:04:38,953-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:04:48,963-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:04:58,975-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:05:01,170-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:05:05,825-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:05:08,986-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:05:18,998-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:05:29,010-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:05:38,488-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:05:39,022-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 13:05:42,300-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 13:05:49,032-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:05:59,041-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:06:09,053-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 13:06:15,807-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 13:06:18,774-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 13:06:19,065-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:06:29,078-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:06:39,087-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:06:49,097-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 13:06:53,124-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 13:06:55,248-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 13:06:59,108-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:07:09,116-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:07:19,125-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:07:29,133-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 13:07:30,443-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:07:31,721-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:07:39,141-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:07:49,149-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:07:59,157-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 13:08:07,762-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 13:08:08,195-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 13:08:09,166-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:08:19,174-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:08:29,183-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:08:39,192-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 13:08:44,670-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 13:08:45,079-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 13:08:49,200-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:08:59,209-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:09:09,217-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:09:19,225-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 13:09:21,145-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 13:09:22,397-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 13:09:29,234-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:09:39,244-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:09:49,253-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:09:57,618-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:09:59,263-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:09:59,715-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:10:09,274-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:10:19,282-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:10:29,292-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-2013-11-06 13:10:34,094-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 13:10:37,033-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 13:10:39,306-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:10:49,317-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:10:59,327-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:11:09,335-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:11:10,566-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:11:14,351-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:11:19,342-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:11:29,351-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:11:39,362-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 13:11:47,041-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 13:11:49,375-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 13:11:51,670-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 13:11:59,387-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:12:09,396-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:12:19,406-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:12:23,514-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:12:28,987-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:12:29,415-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:12:39,426-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:12:49,438-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:12:59,450-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 13:12:59,989-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 13:13:06,306-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 13:13:09,464-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:13:19,479-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:13:29,488-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:13:36,462-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:13:39,501-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 13:13:43,623-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 13:13:49,515-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:13:59,528-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:14:09,540-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 13:14:12,936-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 13:14:19,550-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:14:20,942-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:14:29,555-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:14:39,572-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 13:14:49,411-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 13:14:49,590-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 13:14:58,260-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 13:14:59,608-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:15:09,623-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:15:19,625-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:15:25,885-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:15:29,637-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:15:35,577-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:15:39,649-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:15:49,662-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:15:59,674-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:16:02,358-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:16:09,687-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:16:12,895-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:16:19,698-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:16:29,709-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 13:16:38,833-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 13:16:39,719-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:16:49,730-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 13:16:50,213-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 13:16:59,742-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:17:09,759-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 13:17:15,307-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 13:17:19,772-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 13:17:27,532-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 13:17:29,783-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:17:39,796-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:17:49,808-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:17:51,781-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:17:59,820-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 13:18:04,850-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 13:18:09,832-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:18:19,841-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 13:18:28,255-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 13:18:29,851-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:18:39,862-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:18:42,167-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:18:49,872-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:18:59,881-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:19:04,728-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:19:09,889-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:19:19,485-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:19:19,896-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:19:29,904-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:19:39,912-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:19:41,202-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:19:49,920-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:19:56,803-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:19:59,928-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:20:09,936-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 13:20:17,677-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 13:20:19,944-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:20:29,952-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:20:34,121-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:20:39,960-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:20:49,968-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:20:54,150-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:20:59,976-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:21:09,984-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:21:11,439-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:21:19,992-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:21:30,000-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:21:30,624-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:21:40,008-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 13:21:48,758-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 13:21:50,016-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:22:00,025-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 13:22:07,099-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 13:22:10,035-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:22:20,045-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 13:22:26,076-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 13:22:30,060-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:22:40,062-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 13:22:43,573-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 13:22:50,066-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:23:00,070-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:23:03,394-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:23:10,072-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 13:23:20,047-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 13:23:20,086-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:23:30,098-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:23:40,107-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:23:40,711-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:23:50,117-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-2013-11-06 13:23:56,522-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-No streams
-2013-11-06 13:24:00,132-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:24:10,143-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:24:18,029-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:24:20,155-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:24:30,166-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:24:32,995-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:24:40,179-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:24:50,196-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:24:55,348-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:25:00,199-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:25:09,469-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:25:10,207-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:25:20,215-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:25:30,235-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:25:32,666-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:25:40,247-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:25:45,944-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:25:50,256-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:26:00,266-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:26:09,984-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:26:10,274-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:26:20,283-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 13:26:22,418-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 13:26:30,292-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:26:40,299-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:26:47,302-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:26:50,307-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:26:58,891-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:27:00,320-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:27:10,332-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:27:20,348-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:27:24,620-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:27:30,363-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:27:35,365-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:27:40,372-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:27:50,381-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:28:00,394-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:28:01,938-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:28:10,408-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:28:11,839-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:28:20,423-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:28:30,431-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 13:28:39,256-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 13:28:40,439-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:28:48,313-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:28:50,451-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:29:00,462-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:29:10,480-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:29:16,574-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:29:20,483-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:29:24,788-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:29:30,490-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:29:40,500-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:29:50,520-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:29:53,892-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:30:00,539-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:30:01,262-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:30:10,559-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:30:20,572-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:30:30,581-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:30:31,210-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 13:30:37,737-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 13:30:40,589-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:30:50,600-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:31:00,610-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 13:31:08,529-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 13:31:10,619-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 13:31:14,211-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 13:31:20,627-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:31:30,634-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:31:40,642-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 6ms
-2013-11-06 13:31:45,851-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 6ms
-No streams
-2013-11-06 13:31:50,650-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:31:50,685-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:32:00,657-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:32:10,665-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:32:20,674-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:32:23,164-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:32:27,159-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:32:30,683-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:32:40,689-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:32:50,697-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 9ms
-2013-11-06 13:33:00,490-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 9ms
-No streams
-2013-11-06 13:33:00,706-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 13:33:03,634-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 13:33:10,718-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:33:20,733-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:33:30,745-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:33:37,800-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:33:40,107-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:33:40,763-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:33:50,773-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:34:00,784-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:34:10,795-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:34:15,118-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:34:16,581-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:34:20,804-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:34:30,815-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:34:40,826-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:34:50,836-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 13:34:52,437-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:34:53,055-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:35:00,847-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:35:10,857-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:35:20,866-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 13:35:29,530-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:35:29,754-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:35:30,875-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:35:40,885-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:35:50,894-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:36:00,903-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 13:36:06,004-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:36:07,073-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:36:10,914-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:36:20,924-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:36:30,933-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:36:40,943-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 13:36:42,478-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:36:44,391-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:36:50,952-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:37:00,962-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:37:10,972-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 13:37:18,953-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 13:37:20,981-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 13:37:21,711-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 13:37:30,991-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:37:41,000-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:37:51,010-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:37:55,425-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:37:59,028-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:38:01,020-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:38:11,029-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:38:21,037-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:38:31,049-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:38:31,899-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:38:36,346-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:38:41,054-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:38:51,064-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:39:01,071-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 13:39:08,374-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 13:39:11,079-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:39:13,664-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:39:21,087-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:39:31,107-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:39:41,127-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 3ms
-2013-11-06 13:39:44,849-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 3ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 9ms, rtt: 10ms
-2013-11-06 13:39:50,991-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 9ms, rtt: 10ms
-No streams
-2013-11-06 13:39:51,136-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:40:01,150-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:40:11,162-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:40:21,173-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 13:40:21,322-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 4ms
-2013-11-06 13:40:28,303-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 4ms
-No streams
-2013-11-06 13:40:31,181-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:40:41,188-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:40:51,196-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:40:57,795-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:41:01,203-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:41:05,618-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:41:11,210-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:41:21,218-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:41:31,225-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:41:34,269-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:41:41,233-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:41:42,937-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:41:51,240-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:42:01,248-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:42:10,743-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:42:11,255-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:42:20,255-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:42:21,263-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:42:31,274-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:42:41,282-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:42:47,217-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:42:51,289-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:42:57,573-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:43:01,296-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:43:11,304-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:43:21,311-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:43:23,691-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:43:31,318-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:43:34,891-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:43:41,325-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:43:51,333-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:44:00,165-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:44:01,340-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:44:11,347-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:44:12,209-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:44:21,354-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:44:31,362-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:44:36,639-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:44:41,369-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:44:49,527-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:44:51,377-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:45:01,385-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:45:11,392-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 13:45:13,114-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 13:45:21,400-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 13:45:26,846-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 13:45:31,408-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:45:41,415-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:45:49,587-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:45:51,423-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:46:01,431-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:46:04,163-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:46:11,438-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:46:21,446-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 13:46:26,062-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 13:46:31,454-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:46:41,462-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:46:41,481-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:46:51,469-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:47:01,477-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:47:02,535-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:47:11,485-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:47:18,799-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:47:21,493-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:47:31,500-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:47:39,009-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:47:41,508-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:47:51,516-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:47:56,117-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:48:01,524-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:48:11,532-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:48:15,483-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:48:21,540-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:48:31,547-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 13:48:33,436-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 13:48:41,555-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:48:51,563-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:48:51,957-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:49:01,571-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:49:10,754-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:49:11,579-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:49:21,587-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 13:49:28,432-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 13:49:31,594-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:49:41,602-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:49:48,071-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:49:51,610-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:50:01,618-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:50:04,905-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:50:11,626-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:50:21,634-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 13:50:25,390-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 13:50:31,642-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:50:41,380-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:50:41,650-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:50:51,658-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:51:01,666-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:51:02,707-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:51:11,674-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:51:17,854-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:51:21,681-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:51:31,689-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:51:40,025-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:51:41,697-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:51:51,705-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 13:51:54,329-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 13:52:01,713-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:52:11,721-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:52:17,343-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:52:21,729-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:52:30,803-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:52:31,737-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:52:41,744-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:52:51,752-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:52:54,661-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:53:01,761-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:53:07,277-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:53:11,769-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:53:21,777-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:53:31,785-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 13:53:31,979-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 13:53:41,793-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:53:43,751-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:53:51,801-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:54:01,809-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:54:09,297-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:54:11,817-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 13:54:20,225-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 13:54:21,825-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:54:31,833-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:54:41,841-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:54:46,615-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:54:51,849-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:54:56,699-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:55:01,858-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:55:11,866-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:55:21,874-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 13:55:23,934-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 13:55:31,882-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:55:33,173-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:55:41,891-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:55:51,899-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:56:01,251-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:56:01,907-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:56:09,647-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:56:11,916-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:56:21,924-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:56:31,932-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 13:56:38,569-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 13:56:41,941-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 13:56:46,120-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 13:56:51,949-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:57:01,957-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:57:11,965-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:57:15,887-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:57:21,974-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:57:22,595-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:57:31,982-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:57:41,990-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:57:51,998-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:57:53,205-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 13:57:59,069-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 13:58:02,006-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:58:12,015-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:58:22,023-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:58:30,523-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:58:32,031-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:58:35,543-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:58:42,039-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:58:52,047-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 13:59:02,054-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:59:07,841-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 13:59:12,017-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 13:59:12,062-0600 INFO Multiplexer No streams
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 13:59:12,170-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:40899] for addition
-2013-11-06 13:59:12,171-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:40899] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:40899]
-2013-11-06 13:59:12,171-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:40899]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fe3
-2013-11-06 13:59:12,204-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fe3
-MetaChannel [context: cps-40899, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:40899]
-2013-11-06 13:59:12,204-0600 INFO MetaChannel MetaChannel [context: cps-40899, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:40899]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 13:59:12,204-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 13:59:12,244-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 13:59:12,245-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 13:59:12,325-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 13:59:12,368-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 17.0) submitting urn:job-0-1383759274534-1383759274535: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 13:59:12,368-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 17.0) submitting urn:job-0-1383759274534-1383759274535: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 13:59:12,369-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 13:59:12,369-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-06 13:59:12,375-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7366, JOBSTATUS) unregistering (send)
-2013-11-06 13:59:12,375-0600 INFO RequestHandler Handler(tag: 7366, JOBSTATUS) unregistering (send)
-2013-11-06 13:59:14,449-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 13:59:14,449-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 13:59:14,449-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Handler(tag: 7367, JOBSTATUS) unregistering (send)
-2013-11-06 13:59:14,449-0600 INFO RequestHandler Handler(tag: 7367, JOBSTATUS) unregistering (send)
-2013-11-06 13:59:14,449-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-2013-11-06 13:59:14,449-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 13:59:14,450-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:40900] for addition
-2013-11-06 13:59:14,450-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:40900] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:40900]
-2013-11-06 13:59:14,450-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:40900]
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7fe2
-2013-11-06 13:59:14,472-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7fe2
-MetaChannel [context: cps-40900, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:40900]
-2013-11-06 13:59:14,472-0600 INFO MetaChannel MetaChannel [context: cps-40900, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:40900]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-06 13:59:14,472-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-06 13:59:14,512-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 13:59:14,593-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 13:59:14,599-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 18.0) submitting urn:job-1-1383759274537-1383759274538: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 13:59:14,599-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 18.0) submitting urn:job-1-1383759274537-1383759274538: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 13:59:14,600-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 13:59:14,600-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-06 13:59:14,607-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7368, JOBSTATUS) unregistering (send)
-2013-11-06 13:59:14,607-0600 INFO RequestHandler Handler(tag: 7368, JOBSTATUS) unregistering (send)
-2013-11-06 13:59:16,618-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 13:59:16,618-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 13:59:16,618-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Handler(tag: 7369, JOBSTATUS) unregistering (send)
-2013-11-06 13:59:16,618-0600 INFO RequestHandler Handler(tag: 7369, JOBSTATUS) unregistering (send)
-2013-11-06 13:59:16,618-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-2013-11-06 13:59:16,618-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 13:59:16,619-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:40901] for addition
-2013-11-06 13:59:16,619-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:40901] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:40901]
-2013-11-06 13:59:16,620-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:40901]
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7fe1
-2013-11-06 13:59:16,660-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7fe1
-MetaChannel [context: cps-40901, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:40901]
-2013-11-06 13:59:16,660-0600 INFO MetaChannel MetaChannel [context: cps-40901, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:40901]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-06 13:59:16,660-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-06 13:59:16,680-0600 INFO RequestHandler Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-06 13:59:16,761-0600 INFO RequestHandler Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-06 13:59:16,769-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 19.0) submitting urn:job-2-1383759274540-1383759274541: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 13:59:16,769-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 19.0) submitting urn:job-2-1383759274540-1383759274541: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 13:59:16,769-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 13:59:16,769-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-06 13:59:16,775-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7370, JOBSTATUS) unregistering (send)
-2013-11-06 13:59:16,776-0600 INFO RequestHandler Handler(tag: 7370, JOBSTATUS) unregistering (send)
-2013-11-06 13:59:18,791-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 13:59:18,791-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 13:59:18,791-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-06 13:59:18,791-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7371, JOBSTATUS) unregistering (send)
-2013-11-06 13:59:18,791-0600 INFO RequestHandler Handler(tag: 7371, JOBSTATUS) unregistering (send)
-2013-11-06 13:59:18,791-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-Avg stream buf: 0
-2013-11-06 13:59:22,076-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 13:59:32,086-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 13:59:42,095-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 13:59:45,160-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 13:59:48,492-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 13:59:52,104-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 14:00:02,113-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 14:00:12,122-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 14:00:22,131-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:00:22,477-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:00:24,966-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 14:00:32,141-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 14:00:42,150-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 14:00:52,159-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:00:59,796-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:01:01,440-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 14:01:02,169-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 14:01:12,178-0600 INFO Multiplexer Avg stream buf: 0
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135912.325, now=131106-140113.172, channel=TCPChannel [type: server, contact: 127.0.0.1:40899]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:01:13,172-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135912.325, now=131106-140113.172, channel=TCPChannel [type: server, contact: 127.0.0.1:40899]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 14:01:13,172-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135912.325, now=131106-140113.172, channel=TCPChannel [type: server, contact: 127.0.0.1:40899]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:01:13,172-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135912.325, now=131106-140113.172, channel=TCPChannel [type: server, contact: 127.0.0.1:40899]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 14:01:13,172-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135912.325, now=131106-140113.172, channel=TCPChannel [type: server, contact: 127.0.0.1:40899]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:01:13,172-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135912.325, now=131106-140113.172, channel=TCPChannel [type: server, contact: 127.0.0.1:40899]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 14:01:13,173-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135912.325, now=131106-140113.172, channel=TCPChannel [type: server, contact: 127.0.0.1:40899]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:01:13,173-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135912.325, now=131106-140113.172, channel=TCPChannel [type: server, contact: 127.0.0.1:40899]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 14:01:13,173-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135912.325, now=131106-140113.172, channel=TCPChannel [type: server, contact: 127.0.0.1:40899]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:01:13,173-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135912.325, now=131106-140113.172, channel=TCPChannel [type: server, contact: 127.0.0.1:40899]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 14:01:13,173-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:40899] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135912.325, now=131106-140113.172, channel=TCPChannel [type: server, contact: 127.0.0.1:40899]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:01:13,173-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:40899] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135912.325, now=131106-140113.172, channel=TCPChannel [type: server, contact: 127.0.0.1:40899]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-null: Channel shut down
-2013-11-06 14:01:13,174-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-40899, boundTo: TCPChannel [type: server, contact: 127.0.0.1:40899]] binding to NullChannel
-2013-11-06 14:01:13,174-0600 INFO MetaChannel MetaChannel [context: cps-40899, boundTo: TCPChannel [type: server, contact: 127.0.0.1:40899]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:40899]
-2013-11-06 14:01:13,174-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:40899]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135912.325, now=131106-140113.172, channel=TCPChannel [type: server, contact: 127.0.0.1:40899]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:01:13,174-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135912.325, now=131106-140113.172, channel=TCPChannel [type: server, contact: 127.0.0.1:40899]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 14:01:13,174-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135912.325, now=131106-140113.172, channel=TCPChannel [type: server, contact: 127.0.0.1:40899]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:01:13,174-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135912.325, now=131106-140113.172, channel=TCPChannel [type: server, contact: 127.0.0.1:40899]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 14:01:13,174-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135912.325, now=131106-140113.172, channel=TCPChannel [type: server, contact: 127.0.0.1:40899]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:01:13,175-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135912.325, now=131106-140113.172, channel=TCPChannel [type: server, contact: 127.0.0.1:40899]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 14:01:13,175-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135912.325, now=131106-140113.172, channel=TCPChannel [type: server, contact: 127.0.0.1:40899]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:01:13,175-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135912.325, now=131106-140113.172, channel=TCPChannel [type: server, contact: 127.0.0.1:40899]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 14:01:13,175-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135912.325, now=131106-140113.172, channel=TCPChannel [type: server, contact: 127.0.0.1:40899]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:01:13,175-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135912.325, now=131106-140113.172, channel=TCPChannel [type: server, contact: 127.0.0.1:40899]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel exception handled
-2013-11-06 14:01:13,175-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 14:01:13,181-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 14:01:13,181-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 14:01:13,181-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:40899] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 14:01:13,181-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:40899] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 14:01:13,181-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:40899]
-2013-11-06 14:01:13,182-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:40899]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135914.593, now=131106-140115.450, channel=TCPChannel [type: server, contact: 127.0.0.1:40900]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:01:15,450-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135914.593, now=131106-140115.450, channel=TCPChannel [type: server, contact: 127.0.0.1:40900]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 14:01:15,450-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135914.593, now=131106-140115.450, channel=TCPChannel [type: server, contact: 127.0.0.1:40900]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:01:15,450-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135914.593, now=131106-140115.450, channel=TCPChannel [type: server, contact: 127.0.0.1:40900]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 14:01:15,450-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135914.593, now=131106-140115.450, channel=TCPChannel [type: server, contact: 127.0.0.1:40900]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:01:15,451-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135914.593, now=131106-140115.450, channel=TCPChannel [type: server, contact: 127.0.0.1:40900]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 14:01:15,451-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135914.593, now=131106-140115.450, channel=TCPChannel [type: server, contact: 127.0.0.1:40900]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:01:15,451-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135914.593, now=131106-140115.450, channel=TCPChannel [type: server, contact: 127.0.0.1:40900]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 14:01:15,451-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135914.593, now=131106-140115.450, channel=TCPChannel [type: server, contact: 127.0.0.1:40900]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:01:15,451-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135914.593, now=131106-140115.450, channel=TCPChannel [type: server, contact: 127.0.0.1:40900]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 14:01:15,451-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:40900] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135914.593, now=131106-140115.450, channel=TCPChannel [type: server, contact: 127.0.0.1:40900]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:01:15,451-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:40900] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135914.593, now=131106-140115.450, channel=TCPChannel [type: server, contact: 127.0.0.1:40900]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-null: Channel shut down
-2013-11-06 14:01:15,452-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-40900, boundTo: TCPChannel [type: server, contact: 127.0.0.1:40900]] binding to NullChannel
-2013-11-06 14:01:15,452-0600 INFO MetaChannel MetaChannel [context: cps-40900, boundTo: TCPChannel [type: server, contact: 127.0.0.1:40900]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:40900]
-2013-11-06 14:01:15,452-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:40900]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135914.593, now=131106-140115.450, channel=TCPChannel [type: server, contact: 127.0.0.1:40900]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:01:15,452-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135914.593, now=131106-140115.450, channel=TCPChannel [type: server, contact: 127.0.0.1:40900]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 14:01:15,452-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135914.593, now=131106-140115.450, channel=TCPChannel [type: server, contact: 127.0.0.1:40900]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:01:15,452-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135914.593, now=131106-140115.450, channel=TCPChannel [type: server, contact: 127.0.0.1:40900]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 14:01:15,452-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135914.593, now=131106-140115.450, channel=TCPChannel [type: server, contact: 127.0.0.1:40900]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:01:15,452-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135914.593, now=131106-140115.450, channel=TCPChannel [type: server, contact: 127.0.0.1:40900]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 14:01:15,453-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135914.593, now=131106-140115.450, channel=TCPChannel [type: server, contact: 127.0.0.1:40900]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:01:15,453-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135914.593, now=131106-140115.450, channel=TCPChannel [type: server, contact: 127.0.0.1:40900]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 14:01:15,453-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135914.593, now=131106-140115.450, channel=TCPChannel [type: server, contact: 127.0.0.1:40900]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:01:15,453-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135914.593, now=131106-140115.450, channel=TCPChannel [type: server, contact: 127.0.0.1:40900]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel exception handled
-2013-11-06 14:01:15,453-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 14:01:15,468-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 14:01:15,468-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 14:01:15,469-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:40900] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 14:01:15,469-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:40900] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 14:01:15,469-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:40900]
-2013-11-06 14:01:15,469-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:40900]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135916.760, now=131106-140117.620, channel=TCPChannel [type: server, contact: 127.0.0.1:40901]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:01:17,620-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135916.760, now=131106-140117.620, channel=TCPChannel [type: server, contact: 127.0.0.1:40901]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 14:01:17,620-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135916.760, now=131106-140117.620, channel=TCPChannel [type: server, contact: 127.0.0.1:40901]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:01:17,620-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135916.760, now=131106-140117.620, channel=TCPChannel [type: server, contact: 127.0.0.1:40901]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 14:01:17,620-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135916.760, now=131106-140117.620, channel=TCPChannel [type: server, contact: 127.0.0.1:40901]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:01:17,620-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135916.760, now=131106-140117.620, channel=TCPChannel [type: server, contact: 127.0.0.1:40901]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 14:01:17,621-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135916.760, now=131106-140117.620, channel=TCPChannel [type: server, contact: 127.0.0.1:40901]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:01:17,621-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135916.760, now=131106-140117.620, channel=TCPChannel [type: server, contact: 127.0.0.1:40901]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 14:01:17,621-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135916.760, now=131106-140117.620, channel=TCPChannel [type: server, contact: 127.0.0.1:40901]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:01:17,621-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135916.760, now=131106-140117.620, channel=TCPChannel [type: server, contact: 127.0.0.1:40901]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 14:01:17,621-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:40901] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135916.760, now=131106-140117.620, channel=TCPChannel [type: server, contact: 127.0.0.1:40901]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:01:17,621-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:40901] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135916.760, now=131106-140117.620, channel=TCPChannel [type: server, contact: 127.0.0.1:40901]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-null: Channel shut down
-2013-11-06 14:01:17,622-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-40901, boundTo: TCPChannel [type: server, contact: 127.0.0.1:40901]] binding to NullChannel
-2013-11-06 14:01:17,622-0600 INFO MetaChannel MetaChannel [context: cps-40901, boundTo: TCPChannel [type: server, contact: 127.0.0.1:40901]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:40901]
-2013-11-06 14:01:17,622-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:40901]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135916.760, now=131106-140117.620, channel=TCPChannel [type: server, contact: 127.0.0.1:40901]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:01:17,622-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135916.760, now=131106-140117.620, channel=TCPChannel [type: server, contact: 127.0.0.1:40901]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 14:01:17,622-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135916.760, now=131106-140117.620, channel=TCPChannel [type: server, contact: 127.0.0.1:40901]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:01:17,622-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135916.760, now=131106-140117.620, channel=TCPChannel [type: server, contact: 127.0.0.1:40901]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 14:01:17,622-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135916.760, now=131106-140117.620, channel=TCPChannel [type: server, contact: 127.0.0.1:40901]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:01:17,622-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135916.760, now=131106-140117.620, channel=TCPChannel [type: server, contact: 127.0.0.1:40901]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 14:01:17,622-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135916.760, now=131106-140117.620, channel=TCPChannel [type: server, contact: 127.0.0.1:40901]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:01:17,623-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135916.760, now=131106-140117.620, channel=TCPChannel [type: server, contact: 127.0.0.1:40901]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 14:01:17,623-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135916.760, now=131106-140117.620, channel=TCPChannel [type: server, contact: 127.0.0.1:40901]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:01:17,623-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-135916.760, now=131106-140117.620, channel=TCPChannel [type: server, contact: 127.0.0.1:40901]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel exception handled
-2013-11-06 14:01:17,623-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 14:01:17,635-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 14:01:17,635-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 14:01:17,636-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:40901] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 14:01:17,636-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:40901] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 14:01:17,636-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:40901]
-2013-11-06 14:01:17,636-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:40901]
-No streams
-2013-11-06 14:01:22,189-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:01:32,197-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:01:37,113-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:01:37,914-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 14:01:42,205-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:01:52,213-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:02:02,221-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:02:12,229-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 5ms
-2013-11-06 14:02:14,392-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 5ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:02:14,431-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 14:02:16,727-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:40903] for addition
-2013-11-06 14:02:16,727-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:40903] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:40903]
-2013-11-06 14:02:16,727-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:40903]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fe0
-2013-11-06 14:02:16,762-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fe0
-MetaChannel [context: cps-40903, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:40903]
-2013-11-06 14:02:16,763-0600 INFO MetaChannel MetaChannel [context: cps-40903, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:40903]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 14:02:16,763-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 14:02:16,803-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 14:02:16,803-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 14:02:16,884-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 14:02:16,893-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 20.0) submitting urn:job-0-1383759274543-1383759274544: /bin/echo
-2013-11-06 14:02:16,893-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 20.0) submitting urn:job-0-1383759274543-1383759274544: /bin/echo
-2013-11-06 14:02:16,894-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 14:02:16,894-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-06 14:02:16,900-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7372, JOBSTATUS) unregistering (send)
-2013-11-06 14:02:16,900-0600 INFO RequestHandler Handler(tag: 7372, JOBSTATUS) unregistering (send)
-2013-11-06 14:02:16,919-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 14:02:16,919-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 14:02:16,919-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-06 14:02:16,919-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7373, JOBSTATUS) unregistering (send)
-2013-11-06 14:02:16,919-0600 INFO RequestHandler Handler(tag: 7373, JOBSTATUS) unregistering (send)
-2013-11-06 14:02:16,919-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-Avg stream buf: 0
-2013-11-06 14:02:22,239-0600 INFO Multiplexer Avg stream buf: 0
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 14:02:25,463-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:40904] for addition
-2013-11-06 14:02:25,464-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:40904] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:40904]
-2013-11-06 14:02:25,464-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:40904]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fdf
-2013-11-06 14:02:25,492-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fdf
-MetaChannel [context: cps-40904, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:40904]
-2013-11-06 14:02:25,492-0600 INFO MetaChannel MetaChannel [context: cps-40904, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:40904]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 14:02:25,492-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 14:02:25,533-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 14:02:25,613-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 14:02:25,630-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 21.0) submitting urn:job-0-1383759274546-1383759274547: /bin/echo
-2013-11-06 14:02:25,630-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 21.0) submitting urn:job-0-1383759274546-1383759274547: /bin/echo
-2013-11-06 14:02:25,631-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 14:02:25,631-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-06 14:02:25,638-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7374, JOBSTATUS) unregistering (send)
-2013-11-06 14:02:25,638-0600 INFO RequestHandler Handler(tag: 7374, JOBSTATUS) unregistering (send)
-2013-11-06 14:02:25,640-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 14:02:25,640-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 14:02:25,640-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Handler(tag: 7375, JOBSTATUS) unregistering (send)
-2013-11-06 14:02:25,640-0600 INFO RequestHandler Handler(tag: 7375, JOBSTATUS) unregistering (send)
-2013-11-06 14:02:25,641-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-2013-11-06 14:02:25,641-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-Avg stream buf: 0
-2013-11-06 14:02:32,253-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 14:02:42,262-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 14:02:50,863-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:02:51,749-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 14:02:52,271-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 14:03:02,280-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 14:03:12,288-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 14:03:22,297-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:03:27,336-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:03:29,067-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 14:03:32,307-0600 INFO Multiplexer Avg stream buf: 0
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 14:03:32,483-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:40905] for addition
-2013-11-06 14:03:32,483-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:40905] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:40905]
-2013-11-06 14:03:32,483-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:40905]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fde
-2013-11-06 14:03:32,508-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fde
-MetaChannel [context: cps-40905, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:40905]
-2013-11-06 14:03:32,508-0600 INFO MetaChannel MetaChannel [context: cps-40905, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:40905]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 14:03:32,508-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 14:03:32,549-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 14:03:32,549-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 14:03:32,629-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 14:03:32,671-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 22.0) submitting urn:job-0-1383759274549-1383759274550: /bin/echo
-2013-11-06 14:03:32,671-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 22.0) submitting urn:job-0-1383759274549-1383759274550: /bin/echo
-2013-11-06 14:03:32,672-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 14:03:32,672-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-06 14:03:32,678-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7376, JOBSTATUS) unregistering (send)
-2013-11-06 14:03:32,678-0600 INFO RequestHandler Handler(tag: 7376, JOBSTATUS) unregistering (send)
-2013-11-06 14:03:32,680-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 14:03:32,680-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 14:03:32,680-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Handler(tag: 7377, JOBSTATUS) unregistering (send)
-2013-11-06 14:03:32,681-0600 INFO RequestHandler Handler(tag: 7377, JOBSTATUS) unregistering (send)
-2013-11-06 14:03:32,681-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-2013-11-06 14:03:32,681-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-Avg stream buf: 0
-2013-11-06 14:03:42,318-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 14:03:52,327-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 14:04:02,337-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:04:03,810-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:04:06,385-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 14:04:10,904-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:40906] for addition
-2013-11-06 14:04:10,904-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:40906] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:40906]
-2013-11-06 14:04:10,904-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:40906]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fdd
-2013-11-06 14:04:10,942-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fdd
-MetaChannel [context: cps-40906, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:40906]
-2013-11-06 14:04:10,943-0600 INFO MetaChannel MetaChannel [context: cps-40906, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:40906]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 14:04:10,943-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 14:04:10,963-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 14:04:10,963-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 14:04:11,044-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 14:04:11,075-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 23.0) submitting urn:job-0-1383759274552-1383759274553: /bin/echo
-2013-11-06 14:04:11,075-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 23.0) submitting urn:job-0-1383759274552-1383759274553: /bin/echo
-2013-11-06 14:04:11,076-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 14:04:11,076-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-06 14:04:11,083-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7378, JOBSTATUS) unregistering (send)
-2013-11-06 14:04:11,083-0600 INFO RequestHandler Handler(tag: 7378, JOBSTATUS) unregistering (send)
-2013-11-06 14:04:11,085-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 14:04:11,085-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 14:04:11,085-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Handler(tag: 7379, JOBSTATUS) unregistering (send)
-2013-11-06 14:04:11,085-0600 INFO RequestHandler Handler(tag: 7379, JOBSTATUS) unregistering (send)
-2013-11-06 14:04:11,085-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-2013-11-06 14:04:11,085-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-Avg stream buf: 0
-2013-11-06 14:04:12,348-0600 INFO Multiplexer Avg stream buf: 0
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140216.883, now=131106-140417.727, channel=TCPChannel [type: server, contact: 127.0.0.1:40903]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:04:17,727-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140216.883, now=131106-140417.727, channel=TCPChannel [type: server, contact: 127.0.0.1:40903]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 14:04:17,728-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140216.883, now=131106-140417.727, channel=TCPChannel [type: server, contact: 127.0.0.1:40903]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:04:17,728-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140216.883, now=131106-140417.727, channel=TCPChannel [type: server, contact: 127.0.0.1:40903]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 14:04:17,728-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140216.883, now=131106-140417.727, channel=TCPChannel [type: server, contact: 127.0.0.1:40903]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:04:17,728-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140216.883, now=131106-140417.727, channel=TCPChannel [type: server, contact: 127.0.0.1:40903]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 14:04:17,728-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140216.883, now=131106-140417.727, channel=TCPChannel [type: server, contact: 127.0.0.1:40903]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:04:17,728-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140216.883, now=131106-140417.727, channel=TCPChannel [type: server, contact: 127.0.0.1:40903]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 14:04:17,729-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140216.883, now=131106-140417.727, channel=TCPChannel [type: server, contact: 127.0.0.1:40903]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:04:17,729-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140216.883, now=131106-140417.727, channel=TCPChannel [type: server, contact: 127.0.0.1:40903]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 14:04:17,729-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:40903] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140216.883, now=131106-140417.727, channel=TCPChannel [type: server, contact: 127.0.0.1:40903]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:04:17,729-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:40903] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140216.883, now=131106-140417.727, channel=TCPChannel [type: server, contact: 127.0.0.1:40903]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-null: Channel shut down
-2013-11-06 14:04:17,729-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-40903, boundTo: TCPChannel [type: server, contact: 127.0.0.1:40903]] binding to NullChannel
-2013-11-06 14:04:17,729-0600 INFO MetaChannel MetaChannel [context: cps-40903, boundTo: TCPChannel [type: server, contact: 127.0.0.1:40903]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:40903]
-2013-11-06 14:04:17,729-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:40903]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140216.883, now=131106-140417.727, channel=TCPChannel [type: server, contact: 127.0.0.1:40903]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:04:17,729-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140216.883, now=131106-140417.727, channel=TCPChannel [type: server, contact: 127.0.0.1:40903]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 14:04:17,730-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140216.883, now=131106-140417.727, channel=TCPChannel [type: server, contact: 127.0.0.1:40903]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:04:17,730-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140216.883, now=131106-140417.727, channel=TCPChannel [type: server, contact: 127.0.0.1:40903]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 14:04:17,730-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140216.883, now=131106-140417.727, channel=TCPChannel [type: server, contact: 127.0.0.1:40903]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:04:17,730-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140216.883, now=131106-140417.727, channel=TCPChannel [type: server, contact: 127.0.0.1:40903]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 14:04:17,730-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140216.883, now=131106-140417.727, channel=TCPChannel [type: server, contact: 127.0.0.1:40903]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:04:17,730-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140216.883, now=131106-140417.727, channel=TCPChannel [type: server, contact: 127.0.0.1:40903]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 14:04:17,730-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140216.883, now=131106-140417.727, channel=TCPChannel [type: server, contact: 127.0.0.1:40903]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:04:17,730-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140216.883, now=131106-140417.727, channel=TCPChannel [type: server, contact: 127.0.0.1:40903]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel exception handled
-2013-11-06 14:04:17,731-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 14:04:17,744-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 14:04:17,744-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 14:04:17,744-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:40903] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 14:04:17,744-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:40903] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 14:04:17,744-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:40903]
-2013-11-06 14:04:17,745-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:40903]
-Avg stream buf: 0
-2013-11-06 14:04:22,358-0600 INFO Multiplexer Avg stream buf: 0
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 14:04:23,298-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:40907] for addition
-2013-11-06 14:04:23,298-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:40907] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:40907]
-2013-11-06 14:04:23,299-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:40907]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fdc
-2013-11-06 14:04:23,322-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fdc
-MetaChannel [context: cps-40907, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:40907]
-2013-11-06 14:04:23,322-0600 INFO MetaChannel MetaChannel [context: cps-40907, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:40907]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 14:04:23,322-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 14:04:23,383-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 14:04:23,383-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 14:04:23,464-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 14:04:23,500-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 24.0) submitting urn:job-0-1383759274555-1383759274556: /bin/echo
-2013-11-06 14:04:23,500-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 24.0) submitting urn:job-0-1383759274555-1383759274556: /bin/echo
-2013-11-06 14:04:23,501-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 14:04:23,501-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-06 14:04:23,509-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7380, JOBSTATUS) unregistering (send)
-2013-11-06 14:04:23,509-0600 INFO RequestHandler Handler(tag: 7380, JOBSTATUS) unregistering (send)
-2013-11-06 14:04:23,512-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 14:04:23,512-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 14:04:23,512-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Handler(tag: 7381, JOBSTATUS) unregistering (send)
-2013-11-06 14:04:23,512-0600 INFO RequestHandler Handler(tag: 7381, JOBSTATUS) unregistering (send)
-2013-11-06 14:04:23,512-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-2013-11-06 14:04:23,512-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140225.613, now=131106-140426.464, channel=TCPChannel [type: server, contact: 127.0.0.1:40904]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:04:26,464-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140225.613, now=131106-140426.464, channel=TCPChannel [type: server, contact: 127.0.0.1:40904]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 14:04:26,464-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140225.613, now=131106-140426.464, channel=TCPChannel [type: server, contact: 127.0.0.1:40904]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:04:26,464-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140225.613, now=131106-140426.464, channel=TCPChannel [type: server, contact: 127.0.0.1:40904]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 14:04:26,465-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140225.613, now=131106-140426.464, channel=TCPChannel [type: server, contact: 127.0.0.1:40904]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:04:26,465-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140225.613, now=131106-140426.464, channel=TCPChannel [type: server, contact: 127.0.0.1:40904]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 14:04:26,465-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140225.613, now=131106-140426.464, channel=TCPChannel [type: server, contact: 127.0.0.1:40904]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:04:26,465-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140225.613, now=131106-140426.464, channel=TCPChannel [type: server, contact: 127.0.0.1:40904]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 14:04:26,465-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140225.613, now=131106-140426.464, channel=TCPChannel [type: server, contact: 127.0.0.1:40904]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:04:26,465-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140225.613, now=131106-140426.464, channel=TCPChannel [type: server, contact: 127.0.0.1:40904]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 14:04:26,466-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:40904] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140225.613, now=131106-140426.464, channel=TCPChannel [type: server, contact: 127.0.0.1:40904]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:04:26,466-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:40904] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140225.613, now=131106-140426.464, channel=TCPChannel [type: server, contact: 127.0.0.1:40904]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-null: Channel shut down
-2013-11-06 14:04:26,466-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-40904, boundTo: TCPChannel [type: server, contact: 127.0.0.1:40904]] binding to NullChannel
-2013-11-06 14:04:26,466-0600 INFO MetaChannel MetaChannel [context: cps-40904, boundTo: TCPChannel [type: server, contact: 127.0.0.1:40904]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:40904]
-2013-11-06 14:04:26,466-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:40904]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140225.613, now=131106-140426.464, channel=TCPChannel [type: server, contact: 127.0.0.1:40904]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:04:26,466-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140225.613, now=131106-140426.464, channel=TCPChannel [type: server, contact: 127.0.0.1:40904]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 14:04:26,466-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140225.613, now=131106-140426.464, channel=TCPChannel [type: server, contact: 127.0.0.1:40904]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:04:26,466-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140225.613, now=131106-140426.464, channel=TCPChannel [type: server, contact: 127.0.0.1:40904]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 14:04:26,467-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140225.613, now=131106-140426.464, channel=TCPChannel [type: server, contact: 127.0.0.1:40904]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:04:26,467-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140225.613, now=131106-140426.464, channel=TCPChannel [type: server, contact: 127.0.0.1:40904]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 14:04:26,467-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140225.613, now=131106-140426.464, channel=TCPChannel [type: server, contact: 127.0.0.1:40904]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:04:26,467-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140225.613, now=131106-140426.464, channel=TCPChannel [type: server, contact: 127.0.0.1:40904]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 14:04:26,467-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140225.613, now=131106-140426.464, channel=TCPChannel [type: server, contact: 127.0.0.1:40904]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:04:26,467-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140225.613, now=131106-140426.464, channel=TCPChannel [type: server, contact: 127.0.0.1:40904]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel exception handled
-2013-11-06 14:04:26,467-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 14:04:26,473-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 14:04:26,473-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 14:04:26,474-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:40904] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 14:04:26,474-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:40904] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 14:04:26,474-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:40904]
-2013-11-06 14:04:26,474-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:40904]
-Avg stream buf: 0
-2013-11-06 14:04:32,371-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:04:40,284-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 14:04:42,380-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:04:43,703-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 14:04:52,389-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 14:05:02,399-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 14:05:12,408-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 14:05:16,759-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:05:21,021-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 14:05:22,417-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 14:05:32,426-0600 INFO Multiplexer Avg stream buf: 0
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140332.629, now=131106-140533.483, channel=TCPChannel [type: server, contact: 127.0.0.1:40905]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:05:33,483-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140332.629, now=131106-140533.483, channel=TCPChannel [type: server, contact: 127.0.0.1:40905]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 14:05:33,484-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140332.629, now=131106-140533.483, channel=TCPChannel [type: server, contact: 127.0.0.1:40905]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:05:33,484-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140332.629, now=131106-140533.483, channel=TCPChannel [type: server, contact: 127.0.0.1:40905]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 14:05:33,484-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140332.629, now=131106-140533.483, channel=TCPChannel [type: server, contact: 127.0.0.1:40905]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:05:33,484-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140332.629, now=131106-140533.483, channel=TCPChannel [type: server, contact: 127.0.0.1:40905]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 14:05:33,484-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140332.629, now=131106-140533.483, channel=TCPChannel [type: server, contact: 127.0.0.1:40905]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:05:33,484-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140332.629, now=131106-140533.483, channel=TCPChannel [type: server, contact: 127.0.0.1:40905]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 14:05:33,485-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140332.629, now=131106-140533.483, channel=TCPChannel [type: server, contact: 127.0.0.1:40905]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:05:33,485-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140332.629, now=131106-140533.483, channel=TCPChannel [type: server, contact: 127.0.0.1:40905]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 14:05:33,485-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:40905] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140332.629, now=131106-140533.483, channel=TCPChannel [type: server, contact: 127.0.0.1:40905]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:05:33,485-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:40905] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140332.629, now=131106-140533.483, channel=TCPChannel [type: server, contact: 127.0.0.1:40905]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-null: Channel shut down
-2013-11-06 14:05:33,485-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-40905, boundTo: TCPChannel [type: server, contact: 127.0.0.1:40905]] binding to NullChannel
-2013-11-06 14:05:33,485-0600 INFO MetaChannel MetaChannel [context: cps-40905, boundTo: TCPChannel [type: server, contact: 127.0.0.1:40905]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:40905]
-2013-11-06 14:05:33,485-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:40905]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140332.629, now=131106-140533.483, channel=TCPChannel [type: server, contact: 127.0.0.1:40905]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:05:33,485-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140332.629, now=131106-140533.483, channel=TCPChannel [type: server, contact: 127.0.0.1:40905]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 14:05:33,486-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140332.629, now=131106-140533.483, channel=TCPChannel [type: server, contact: 127.0.0.1:40905]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:05:33,486-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140332.629, now=131106-140533.483, channel=TCPChannel [type: server, contact: 127.0.0.1:40905]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 14:05:33,486-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140332.629, now=131106-140533.483, channel=TCPChannel [type: server, contact: 127.0.0.1:40905]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:05:33,486-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140332.629, now=131106-140533.483, channel=TCPChannel [type: server, contact: 127.0.0.1:40905]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 14:05:33,486-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140332.629, now=131106-140533.483, channel=TCPChannel [type: server, contact: 127.0.0.1:40905]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:05:33,486-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140332.629, now=131106-140533.483, channel=TCPChannel [type: server, contact: 127.0.0.1:40905]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 14:05:33,486-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140332.629, now=131106-140533.483, channel=TCPChannel [type: server, contact: 127.0.0.1:40905]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:05:33,486-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140332.629, now=131106-140533.483, channel=TCPChannel [type: server, contact: 127.0.0.1:40905]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel exception handled
-2013-11-06 14:05:33,487-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 14:05:33,490-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 14:05:33,490-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 14:05:33,490-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:40905] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 14:05:33,490-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:40905] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 14:05:33,490-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:40905]
-2013-11-06 14:05:33,490-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:40905]
-Avg stream buf: 0
-2013-11-06 14:05:42,436-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 14:05:52,445-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 14:05:53,233-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:05:58,339-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 14:06:02,454-0600 INFO Multiplexer Avg stream buf: 0
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140411.044, now=131106-140611.904, channel=TCPChannel [type: server, contact: 127.0.0.1:40906]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:06:11,904-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140411.044, now=131106-140611.904, channel=TCPChannel [type: server, contact: 127.0.0.1:40906]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 14:06:11,904-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140411.044, now=131106-140611.904, channel=TCPChannel [type: server, contact: 127.0.0.1:40906]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:06:11,904-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140411.044, now=131106-140611.904, channel=TCPChannel [type: server, contact: 127.0.0.1:40906]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 14:06:11,904-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140411.044, now=131106-140611.904, channel=TCPChannel [type: server, contact: 127.0.0.1:40906]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:06:11,904-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140411.044, now=131106-140611.904, channel=TCPChannel [type: server, contact: 127.0.0.1:40906]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 14:06:11,905-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140411.044, now=131106-140611.904, channel=TCPChannel [type: server, contact: 127.0.0.1:40906]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:06:11,905-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140411.044, now=131106-140611.904, channel=TCPChannel [type: server, contact: 127.0.0.1:40906]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 14:06:11,905-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140411.044, now=131106-140611.904, channel=TCPChannel [type: server, contact: 127.0.0.1:40906]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:06:11,905-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140411.044, now=131106-140611.904, channel=TCPChannel [type: server, contact: 127.0.0.1:40906]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 14:06:11,905-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:40906] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140411.044, now=131106-140611.904, channel=TCPChannel [type: server, contact: 127.0.0.1:40906]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:06:11,905-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:40906] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140411.044, now=131106-140611.904, channel=TCPChannel [type: server, contact: 127.0.0.1:40906]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-null: Channel shut down
-2013-11-06 14:06:11,905-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-40906, boundTo: TCPChannel [type: server, contact: 127.0.0.1:40906]] binding to NullChannel
-2013-11-06 14:06:11,906-0600 INFO MetaChannel MetaChannel [context: cps-40906, boundTo: TCPChannel [type: server, contact: 127.0.0.1:40906]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:40906]
-2013-11-06 14:06:11,906-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:40906]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140411.044, now=131106-140611.904, channel=TCPChannel [type: server, contact: 127.0.0.1:40906]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:06:11,906-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140411.044, now=131106-140611.904, channel=TCPChannel [type: server, contact: 127.0.0.1:40906]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 14:06:11,906-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140411.044, now=131106-140611.904, channel=TCPChannel [type: server, contact: 127.0.0.1:40906]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:06:11,906-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140411.044, now=131106-140611.904, channel=TCPChannel [type: server, contact: 127.0.0.1:40906]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 14:06:11,906-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140411.044, now=131106-140611.904, channel=TCPChannel [type: server, contact: 127.0.0.1:40906]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:06:11,906-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140411.044, now=131106-140611.904, channel=TCPChannel [type: server, contact: 127.0.0.1:40906]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 14:06:11,906-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140411.044, now=131106-140611.904, channel=TCPChannel [type: server, contact: 127.0.0.1:40906]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:06:11,906-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140411.044, now=131106-140611.904, channel=TCPChannel [type: server, contact: 127.0.0.1:40906]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 14:06:11,907-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140411.044, now=131106-140611.904, channel=TCPChannel [type: server, contact: 127.0.0.1:40906]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:06:11,907-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140411.044, now=131106-140611.904, channel=TCPChannel [type: server, contact: 127.0.0.1:40906]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel exception handled
-2013-11-06 14:06:11,907-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 14:06:11,921-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 14:06:11,921-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 14:06:11,922-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:40906] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 14:06:11,922-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:40906] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 14:06:11,922-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:40906]
-2013-11-06 14:06:11,922-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:40906]
-Avg stream buf: 0
-2013-11-06 14:06:12,464-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 14:06:22,472-0600 INFO Multiplexer Avg stream buf: 0
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140423.464, now=131106-140624.298, channel=TCPChannel [type: server, contact: 127.0.0.1:40907]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:06:24,298-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140423.464, now=131106-140624.298, channel=TCPChannel [type: server, contact: 127.0.0.1:40907]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 14:06:24,298-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140423.464, now=131106-140624.298, channel=TCPChannel [type: server, contact: 127.0.0.1:40907]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:06:24,298-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140423.464, now=131106-140624.298, channel=TCPChannel [type: server, contact: 127.0.0.1:40907]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 14:06:24,299-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140423.464, now=131106-140624.298, channel=TCPChannel [type: server, contact: 127.0.0.1:40907]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:06:24,299-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140423.464, now=131106-140624.298, channel=TCPChannel [type: server, contact: 127.0.0.1:40907]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 14:06:24,299-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140423.464, now=131106-140624.298, channel=TCPChannel [type: server, contact: 127.0.0.1:40907]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:06:24,299-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140423.464, now=131106-140624.298, channel=TCPChannel [type: server, contact: 127.0.0.1:40907]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 14:06:24,299-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140423.464, now=131106-140624.298, channel=TCPChannel [type: server, contact: 127.0.0.1:40907]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:06:24,299-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140423.464, now=131106-140624.298, channel=TCPChannel [type: server, contact: 127.0.0.1:40907]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 14:06:24,299-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:40907] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140423.464, now=131106-140624.298, channel=TCPChannel [type: server, contact: 127.0.0.1:40907]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:06:24,299-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:40907] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140423.464, now=131106-140624.298, channel=TCPChannel [type: server, contact: 127.0.0.1:40907]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-null: Channel shut down
-2013-11-06 14:06:24,300-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-40907, boundTo: TCPChannel [type: server, contact: 127.0.0.1:40907]] binding to NullChannel
-2013-11-06 14:06:24,300-0600 INFO MetaChannel MetaChannel [context: cps-40907, boundTo: TCPChannel [type: server, contact: 127.0.0.1:40907]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:40907]
-2013-11-06 14:06:24,300-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:40907]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140423.464, now=131106-140624.298, channel=TCPChannel [type: server, contact: 127.0.0.1:40907]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:06:24,300-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140423.464, now=131106-140624.298, channel=TCPChannel [type: server, contact: 127.0.0.1:40907]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 14:06:24,300-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140423.464, now=131106-140624.298, channel=TCPChannel [type: server, contact: 127.0.0.1:40907]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:06:24,300-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140423.464, now=131106-140624.298, channel=TCPChannel [type: server, contact: 127.0.0.1:40907]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 14:06:24,300-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140423.464, now=131106-140624.298, channel=TCPChannel [type: server, contact: 127.0.0.1:40907]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:06:24,300-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140423.464, now=131106-140624.298, channel=TCPChannel [type: server, contact: 127.0.0.1:40907]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 14:06:24,301-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140423.464, now=131106-140624.298, channel=TCPChannel [type: server, contact: 127.0.0.1:40907]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:06:24,301-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140423.464, now=131106-140624.298, channel=TCPChannel [type: server, contact: 127.0.0.1:40907]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 14:06:24,301-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140423.464, now=131106-140624.298, channel=TCPChannel [type: server, contact: 127.0.0.1:40907]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:06:24,301-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140423.464, now=131106-140624.298, channel=TCPChannel [type: server, contact: 127.0.0.1:40907]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel exception handled
-2013-11-06 14:06:24,301-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 14:06:24,318-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 14:06:24,318-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 14:06:24,318-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:40907] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 14:06:24,318-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:40907] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 14:06:24,318-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:40907]
-2013-11-06 14:06:24,318-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:40907]
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:06:29,706-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 14:06:32,481-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:06:35,657-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 14:06:42,489-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:06:52,497-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:07:02,506-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:07:06,180-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 14:07:12,514-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:07:12,975-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 14:07:22,522-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:07:32,530-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:07:42,539-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:07:42,654-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 14:07:50,294-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 14:07:52,548-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:08:02,557-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:08:12,566-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:08:19,128-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 14:08:22,575-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 14:08:27,612-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 14:08:32,590-0600 INFO Multiplexer No streams
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 14:08:41,336-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:40918] for addition
-2013-11-06 14:08:41,336-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:40918] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:40918]
-2013-11-06 14:08:41,336-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:40918]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fdb
-2013-11-06 14:08:41,359-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fdb
-MetaChannel [context: cps-40918, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:40918]
-2013-11-06 14:08:41,359-0600 INFO MetaChannel MetaChannel [context: cps-40918, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:40918]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 14:08:41,359-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 14:08:41,400-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 14:08:41,400-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 14:08:41,481-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 14:08:41,511-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 25.0) submitting urn:job-0-1383759274558-1383759274559: /bin/echo
-2013-11-06 14:08:41,511-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 25.0) submitting urn:job-0-1383759274558-1383759274559: /bin/echo
-2013-11-06 14:08:41,512-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 14:08:41,512-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-06 14:08:41,519-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7382, JOBSTATUS) unregistering (send)
-2013-11-06 14:08:41,519-0600 INFO RequestHandler Handler(tag: 7382, JOBSTATUS) unregistering (send)
-2013-11-06 14:08:41,522-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 14:08:41,522-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 14:08:41,522-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-06 14:08:41,522-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7383, JOBSTATUS) unregistering (send)
-2013-11-06 14:08:41,522-0600 INFO RequestHandler Handler(tag: 7383, JOBSTATUS) unregistering (send)
-2013-11-06 14:08:41,522-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-Avg stream buf: 0
-2013-11-06 14:08:42,605-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 14:08:52,612-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 14:08:55,604-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-Avg stream buf: 0
-2013-11-06 14:09:02,622-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:09:04,929-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 14:09:12,635-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 14:09:22,649-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:09:32,078-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 14:09:32,665-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:09:42,247-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 14:09:42,680-0600 INFO Multiplexer Avg stream buf: 0
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 14:09:48,612-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:40919] for addition
-2013-11-06 14:09:48,613-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:40919] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:40919]
-2013-11-06 14:09:48,613-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:40919]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fda
-2013-11-06 14:09:48,645-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fda
-MetaChannel [context: cps-40919, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:40919]
-2013-11-06 14:09:48,645-0600 INFO MetaChannel MetaChannel [context: cps-40919, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:40919]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 14:09:48,645-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 14:09:48,705-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 14:09:48,705-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 14:09:48,786-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 14:09:48,817-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 26.0) submitting urn:job-0-1383759274561-1383759274562: /bin/echo
-2013-11-06 14:09:48,817-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 26.0) submitting urn:job-0-1383759274561-1383759274562: /bin/echo
-2013-11-06 14:09:48,818-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 14:09:48,818-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-06 14:09:48,827-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7384, JOBSTATUS) unregistering (send)
-2013-11-06 14:09:48,827-0600 INFO RequestHandler Handler(tag: 7384, JOBSTATUS) unregistering (send)
-2013-11-06 14:09:48,829-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 14:09:48,829-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 14:09:48,829-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Handler(tag: 7385, JOBSTATUS) unregistering (send)
-2013-11-06 14:09:48,829-0600 INFO RequestHandler Handler(tag: 7385, JOBSTATUS) unregistering (send)
-2013-11-06 14:09:48,829-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-2013-11-06 14:09:48,829-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-Avg stream buf: 0
-2013-11-06 14:09:52,682-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 14:10:02,696-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:10:08,551-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 14:10:12,706-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 14:10:19,566-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-Avg stream buf: 0
-2013-11-06 14:10:22,722-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 14:10:32,731-0600 INFO Multiplexer Avg stream buf: 0
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140841.480, now=131106-141042.336, channel=TCPChannel [type: server, contact: 127.0.0.1:40918]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:10:42,336-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140841.480, now=131106-141042.336, channel=TCPChannel [type: server, contact: 127.0.0.1:40918]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 14:10:42,336-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140841.480, now=131106-141042.336, channel=TCPChannel [type: server, contact: 127.0.0.1:40918]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:10:42,336-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140841.480, now=131106-141042.336, channel=TCPChannel [type: server, contact: 127.0.0.1:40918]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 14:10:42,337-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140841.480, now=131106-141042.336, channel=TCPChannel [type: server, contact: 127.0.0.1:40918]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:10:42,337-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140841.480, now=131106-141042.336, channel=TCPChannel [type: server, contact: 127.0.0.1:40918]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 14:10:42,337-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140841.480, now=131106-141042.336, channel=TCPChannel [type: server, contact: 127.0.0.1:40918]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:10:42,337-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140841.480, now=131106-141042.336, channel=TCPChannel [type: server, contact: 127.0.0.1:40918]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 14:10:42,337-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140841.480, now=131106-141042.336, channel=TCPChannel [type: server, contact: 127.0.0.1:40918]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:10:42,337-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140841.480, now=131106-141042.336, channel=TCPChannel [type: server, contact: 127.0.0.1:40918]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 14:10:42,338-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:40918] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140841.480, now=131106-141042.336, channel=TCPChannel [type: server, contact: 127.0.0.1:40918]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:10:42,338-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:40918] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140841.480, now=131106-141042.336, channel=TCPChannel [type: server, contact: 127.0.0.1:40918]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-null: Channel shut down
-2013-11-06 14:10:42,338-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-40918, boundTo: TCPChannel [type: server, contact: 127.0.0.1:40918]] binding to NullChannel
-2013-11-06 14:10:42,338-0600 INFO MetaChannel MetaChannel [context: cps-40918, boundTo: TCPChannel [type: server, contact: 127.0.0.1:40918]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:40918]
-2013-11-06 14:10:42,338-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:40918]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140841.480, now=131106-141042.336, channel=TCPChannel [type: server, contact: 127.0.0.1:40918]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:10:42,338-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140841.480, now=131106-141042.336, channel=TCPChannel [type: server, contact: 127.0.0.1:40918]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 14:10:42,339-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140841.480, now=131106-141042.336, channel=TCPChannel [type: server, contact: 127.0.0.1:40918]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:10:42,339-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140841.480, now=131106-141042.336, channel=TCPChannel [type: server, contact: 127.0.0.1:40918]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 14:10:42,339-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140841.480, now=131106-141042.336, channel=TCPChannel [type: server, contact: 127.0.0.1:40918]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:10:42,339-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140841.480, now=131106-141042.336, channel=TCPChannel [type: server, contact: 127.0.0.1:40918]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 14:10:42,339-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140841.480, now=131106-141042.336, channel=TCPChannel [type: server, contact: 127.0.0.1:40918]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:10:42,339-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140841.480, now=131106-141042.336, channel=TCPChannel [type: server, contact: 127.0.0.1:40918]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 14:10:42,339-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140841.480, now=131106-141042.336, channel=TCPChannel [type: server, contact: 127.0.0.1:40918]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:10:42,339-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140841.480, now=131106-141042.336, channel=TCPChannel [type: server, contact: 127.0.0.1:40918]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel exception handled
-2013-11-06 14:10:42,340-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 14:10:42,340-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 14:10:42,341-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 14:10:42,341-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:40918] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 14:10:42,341-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:40918] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 14:10:42,341-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:40918]
-2013-11-06 14:10:42,341-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:40918]
-Avg stream buf: 0
-2013-11-06 14:10:42,743-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 14:10:45,026-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-Avg stream buf: 0
-2013-11-06 14:10:52,752-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:10:56,884-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 14:11:02,762-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 14:11:12,772-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:11:21,500-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 14:11:22,781-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 14:11:32,792-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:11:34,202-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 14:11:42,802-0600 INFO Multiplexer Avg stream buf: 0
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140948.786, now=131106-141149.613, channel=TCPChannel [type: server, contact: 127.0.0.1:40919]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:11:49,613-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140948.786, now=131106-141149.613, channel=TCPChannel [type: server, contact: 127.0.0.1:40919]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 14:11:49,613-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140948.786, now=131106-141149.613, channel=TCPChannel [type: server, contact: 127.0.0.1:40919]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:11:49,613-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140948.786, now=131106-141149.613, channel=TCPChannel [type: server, contact: 127.0.0.1:40919]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 14:11:49,614-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140948.786, now=131106-141149.613, channel=TCPChannel [type: server, contact: 127.0.0.1:40919]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:11:49,614-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140948.786, now=131106-141149.613, channel=TCPChannel [type: server, contact: 127.0.0.1:40919]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 14:11:49,614-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140948.786, now=131106-141149.613, channel=TCPChannel [type: server, contact: 127.0.0.1:40919]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:11:49,614-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140948.786, now=131106-141149.613, channel=TCPChannel [type: server, contact: 127.0.0.1:40919]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 14:11:49,614-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140948.786, now=131106-141149.613, channel=TCPChannel [type: server, contact: 127.0.0.1:40919]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:11:49,614-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140948.786, now=131106-141149.613, channel=TCPChannel [type: server, contact: 127.0.0.1:40919]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 14:11:49,615-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:40919] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140948.786, now=131106-141149.613, channel=TCPChannel [type: server, contact: 127.0.0.1:40919]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:11:49,615-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:40919] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140948.786, now=131106-141149.613, channel=TCPChannel [type: server, contact: 127.0.0.1:40919]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-null: Channel shut down
-2013-11-06 14:11:49,615-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-40919, boundTo: TCPChannel [type: server, contact: 127.0.0.1:40919]] binding to NullChannel
-2013-11-06 14:11:49,615-0600 INFO MetaChannel MetaChannel [context: cps-40919, boundTo: TCPChannel [type: server, contact: 127.0.0.1:40919]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:40919]
-2013-11-06 14:11:49,615-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:40919]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140948.786, now=131106-141149.613, channel=TCPChannel [type: server, contact: 127.0.0.1:40919]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:11:49,615-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140948.786, now=131106-141149.613, channel=TCPChannel [type: server, contact: 127.0.0.1:40919]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 14:11:49,616-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140948.786, now=131106-141149.613, channel=TCPChannel [type: server, contact: 127.0.0.1:40919]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:11:49,616-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140948.786, now=131106-141149.613, channel=TCPChannel [type: server, contact: 127.0.0.1:40919]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 14:11:49,616-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140948.786, now=131106-141149.613, channel=TCPChannel [type: server, contact: 127.0.0.1:40919]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:11:49,616-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140948.786, now=131106-141149.613, channel=TCPChannel [type: server, contact: 127.0.0.1:40919]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 14:11:49,616-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140948.786, now=131106-141149.613, channel=TCPChannel [type: server, contact: 127.0.0.1:40919]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:11:49,616-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140948.786, now=131106-141149.613, channel=TCPChannel [type: server, contact: 127.0.0.1:40919]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 14:11:49,616-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140948.786, now=131106-141149.613, channel=TCPChannel [type: server, contact: 127.0.0.1:40919]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:11:49,616-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-140948.786, now=131106-141149.613, channel=TCPChannel [type: server, contact: 127.0.0.1:40919]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel exception handled
-2013-11-06 14:11:49,617-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 14:11:49,622-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 14:11:49,623-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 14:11:49,623-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:40919] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 14:11:49,623-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:40919] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 14:11:49,623-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:40919]
-2013-11-06 14:11:49,623-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:40919]
-No streams
-2013-11-06 14:11:52,813-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:11:57,973-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 14:12:02,822-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:12:11,521-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 14:12:12,830-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:12:22,840-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:12:32,848-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 14:12:34,448-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 14:12:42,858-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:12:48,839-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 14:12:52,869-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:13:02,877-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:13:10,921-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 14:13:12,886-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:13:22,893-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:13:26,156-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 14:13:32,901-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:13:42,909-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:13:47,396-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 14:13:52,916-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:14:02,924-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:14:03,475-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 14:14:12,932-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:14:22,939-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:14:23,870-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 14:14:32,947-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:14:40,793-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 14:14:42,955-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:14:52,963-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 14:15:00,344-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 14:15:02,970-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:15:12,978-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:15:18,111-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 14:15:22,986-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:15:32,993-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 14:15:36,819-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 14:15:43,001-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:15:53,009-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:15:55,429-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 14:16:03,017-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:16:13,024-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:16:13,293-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 14:16:23,032-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:16:32,747-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 14:16:33,040-0600 INFO Multiplexer No streams
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 14:16:37,767-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:40922] for addition
-2013-11-06 14:16:37,767-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:40922] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:40922]
-2013-11-06 14:16:37,767-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:40922]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fd9
-2013-11-06 14:16:37,794-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fd9
-MetaChannel [context: cps-40922, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:40922]
-2013-11-06 14:16:37,794-0600 INFO MetaChannel MetaChannel [context: cps-40922, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:40922]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 14:16:37,794-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 14:16:37,835-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 14:16:37,835-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 14:16:37,915-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 14:16:37,933-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 27.0) submitting urn:job-0-1383759274564-1383759274565: /bin/echo
-2013-11-06 14:16:37,933-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 27.0) submitting urn:job-0-1383759274564-1383759274565: /bin/echo
-2013-11-06 14:16:37,933-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 14:16:37,933-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-06 14:16:37,942-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7386, JOBSTATUS) unregistering (send)
-2013-11-06 14:16:37,943-0600 INFO RequestHandler Handler(tag: 7386, JOBSTATUS) unregistering (send)
-2013-11-06 14:16:37,945-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 14:16:37,945-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 14:16:37,945-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-06 14:16:37,945-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7387, JOBSTATUS) unregistering (send)
-2013-11-06 14:16:37,945-0600 INFO RequestHandler Handler(tag: 7387, JOBSTATUS) unregistering (send)
-2013-11-06 14:16:37,945-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-Avg stream buf: 0
-2013-11-06 14:16:43,050-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:16:49,766-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 14:16:53,059-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 14:17:03,067-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:17:10,065-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 14:17:13,076-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 14:17:23,084-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:17:26,240-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 14:17:33,093-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 14:17:43,101-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:17:47,383-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 14:17:53,110-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:18:02,714-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 14:18:03,118-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 14:18:13,127-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 14:18:23,136-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:18:24,701-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 14:18:33,144-0600 INFO Multiplexer Avg stream buf: 0
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-141637.915, now=131106-141838.767, channel=TCPChannel [type: server, contact: 127.0.0.1:40922]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:18:38,767-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-141637.915, now=131106-141838.767, channel=TCPChannel [type: server, contact: 127.0.0.1:40922]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 14:18:38,767-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-141637.915, now=131106-141838.767, channel=TCPChannel [type: server, contact: 127.0.0.1:40922]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:18:38,767-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-141637.915, now=131106-141838.767, channel=TCPChannel [type: server, contact: 127.0.0.1:40922]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 14:18:38,767-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-141637.915, now=131106-141838.767, channel=TCPChannel [type: server, contact: 127.0.0.1:40922]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:18:38,767-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-141637.915, now=131106-141838.767, channel=TCPChannel [type: server, contact: 127.0.0.1:40922]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 14:18:38,768-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-141637.915, now=131106-141838.767, channel=TCPChannel [type: server, contact: 127.0.0.1:40922]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:18:38,768-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-141637.915, now=131106-141838.767, channel=TCPChannel [type: server, contact: 127.0.0.1:40922]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 14:18:38,768-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-141637.915, now=131106-141838.767, channel=TCPChannel [type: server, contact: 127.0.0.1:40922]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:18:38,768-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-141637.915, now=131106-141838.767, channel=TCPChannel [type: server, contact: 127.0.0.1:40922]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 14:18:38,768-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:40922] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-141637.915, now=131106-141838.767, channel=TCPChannel [type: server, contact: 127.0.0.1:40922]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:18:38,768-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:40922] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-141637.915, now=131106-141838.767, channel=TCPChannel [type: server, contact: 127.0.0.1:40922]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-null: Channel shut down
-2013-11-06 14:18:38,769-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-40922, boundTo: TCPChannel [type: server, contact: 127.0.0.1:40922]] binding to NullChannel
-2013-11-06 14:18:38,769-0600 INFO MetaChannel MetaChannel [context: cps-40922, boundTo: TCPChannel [type: server, contact: 127.0.0.1:40922]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:40922]
-2013-11-06 14:18:38,769-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:40922]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-141637.915, now=131106-141838.767, channel=TCPChannel [type: server, contact: 127.0.0.1:40922]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:18:38,769-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-141637.915, now=131106-141838.767, channel=TCPChannel [type: server, contact: 127.0.0.1:40922]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 14:18:38,769-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-141637.915, now=131106-141838.767, channel=TCPChannel [type: server, contact: 127.0.0.1:40922]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:18:38,769-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-141637.915, now=131106-141838.767, channel=TCPChannel [type: server, contact: 127.0.0.1:40922]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 14:18:38,769-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-141637.915, now=131106-141838.767, channel=TCPChannel [type: server, contact: 127.0.0.1:40922]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:18:38,769-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-141637.915, now=131106-141838.767, channel=TCPChannel [type: server, contact: 127.0.0.1:40922]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 14:18:38,770-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-141637.915, now=131106-141838.767, channel=TCPChannel [type: server, contact: 127.0.0.1:40922]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:18:38,770-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-141637.915, now=131106-141838.767, channel=TCPChannel [type: server, contact: 127.0.0.1:40922]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 14:18:38,770-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-141637.915, now=131106-141838.767, channel=TCPChannel [type: server, contact: 127.0.0.1:40922]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:18:38,770-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-141637.915, now=131106-141838.767, channel=TCPChannel [type: server, contact: 127.0.0.1:40922]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel exception handled
-2013-11-06 14:18:38,770-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 14:18:38,781-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 14:18:38,781-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 14:18:38,781-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:40922] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 14:18:38,781-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:40922] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 14:18:38,781-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:40922]
-2013-11-06 14:18:38,781-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:40922]
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:18:39,188-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 14:18:43,154-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:18:53,162-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:19:02,019-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 14:19:03,170-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:19:13,179-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:19:15,662-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 14:19:23,187-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:19:33,195-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:19:39,337-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 14:19:43,203-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:19:52,137-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 14:19:53,211-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:20:03,220-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:20:13,228-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:20:16,655-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 14:20:23,236-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:20:28,610-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 14:20:33,245-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:20:43,253-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:20:53,261-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:20:53,973-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 14:21:03,269-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 14:21:05,085-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 14:21:07,045-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:40923] for addition
-2013-11-06 14:21:07,045-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:40923] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:40923]
-2013-11-06 14:21:07,045-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:40923]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fd8
-2013-11-06 14:21:07,081-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fd8
-MetaChannel [context: cps-40923, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:40923]
-2013-11-06 14:21:07,081-0600 INFO MetaChannel MetaChannel [context: cps-40923, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:40923]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 14:21:07,081-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 14:21:07,122-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 14:21:07,122-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 14:21:07,202-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 14:21:07,252-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 28.0) submitting urn:job-0-1383759274567-1383759274568: /bin/echo
-2013-11-06 14:21:07,253-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 28.0) submitting urn:job-0-1383759274567-1383759274568: /bin/echo
-2013-11-06 14:21:07,254-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 14:21:07,254-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-06 14:21:07,260-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7388, JOBSTATUS) unregistering (send)
-2013-11-06 14:21:07,260-0600 INFO RequestHandler Handler(tag: 7388, JOBSTATUS) unregistering (send)
-2013-11-06 14:21:07,262-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 14:21:07,262-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 14:21:07,262-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-06 14:21:07,262-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7389, JOBSTATUS) unregistering (send)
-2013-11-06 14:21:07,262-0600 INFO RequestHandler Handler(tag: 7389, JOBSTATUS) unregistering (send)
-2013-11-06 14:21:07,262-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-Avg stream buf: 0
-2013-11-06 14:21:13,280-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 14:21:23,289-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 14:21:31,292-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-Avg stream buf: 0
-2013-11-06 14:21:33,297-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:21:41,558-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 14:21:43,305-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 14:21:53,314-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 14:22:03,322-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:22:08,609-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 14:22:13,330-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 14:22:18,033-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-Avg stream buf: 0
-2013-11-06 14:22:23,338-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 14:22:33,346-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 14:22:43,354-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:22:45,927-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 14:22:53,363-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:22:54,506-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 14:23:03,371-0600 INFO Multiplexer Avg stream buf: 0
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142107.202, now=131106-142308.045, channel=TCPChannel [type: server, contact: 127.0.0.1:40923]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:23:08,045-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142107.202, now=131106-142308.045, channel=TCPChannel [type: server, contact: 127.0.0.1:40923]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 14:23:08,046-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142107.202, now=131106-142308.045, channel=TCPChannel [type: server, contact: 127.0.0.1:40923]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:23:08,046-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142107.202, now=131106-142308.045, channel=TCPChannel [type: server, contact: 127.0.0.1:40923]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 14:23:08,046-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142107.202, now=131106-142308.045, channel=TCPChannel [type: server, contact: 127.0.0.1:40923]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:23:08,046-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142107.202, now=131106-142308.045, channel=TCPChannel [type: server, contact: 127.0.0.1:40923]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 14:23:08,047-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142107.202, now=131106-142308.045, channel=TCPChannel [type: server, contact: 127.0.0.1:40923]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:23:08,047-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142107.202, now=131106-142308.045, channel=TCPChannel [type: server, contact: 127.0.0.1:40923]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 14:23:08,047-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142107.202, now=131106-142308.045, channel=TCPChannel [type: server, contact: 127.0.0.1:40923]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:23:08,047-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142107.202, now=131106-142308.045, channel=TCPChannel [type: server, contact: 127.0.0.1:40923]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 14:23:08,047-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:40923] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142107.202, now=131106-142308.045, channel=TCPChannel [type: server, contact: 127.0.0.1:40923]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:23:08,047-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:40923] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142107.202, now=131106-142308.045, channel=TCPChannel [type: server, contact: 127.0.0.1:40923]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-null: Channel shut down
-2013-11-06 14:23:08,048-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-40923, boundTo: TCPChannel [type: server, contact: 127.0.0.1:40923]] binding to NullChannel
-2013-11-06 14:23:08,048-0600 INFO MetaChannel MetaChannel [context: cps-40923, boundTo: TCPChannel [type: server, contact: 127.0.0.1:40923]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:40923]
-2013-11-06 14:23:08,048-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:40923]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142107.202, now=131106-142308.045, channel=TCPChannel [type: server, contact: 127.0.0.1:40923]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:23:08,048-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142107.202, now=131106-142308.045, channel=TCPChannel [type: server, contact: 127.0.0.1:40923]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 14:23:08,048-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142107.202, now=131106-142308.045, channel=TCPChannel [type: server, contact: 127.0.0.1:40923]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:23:08,048-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142107.202, now=131106-142308.045, channel=TCPChannel [type: server, contact: 127.0.0.1:40923]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 14:23:08,048-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142107.202, now=131106-142308.045, channel=TCPChannel [type: server, contact: 127.0.0.1:40923]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:23:08,048-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142107.202, now=131106-142308.045, channel=TCPChannel [type: server, contact: 127.0.0.1:40923]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 14:23:08,048-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142107.202, now=131106-142308.045, channel=TCPChannel [type: server, contact: 127.0.0.1:40923]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:23:08,049-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142107.202, now=131106-142308.045, channel=TCPChannel [type: server, contact: 127.0.0.1:40923]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 14:23:08,049-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142107.202, now=131106-142308.045, channel=TCPChannel [type: server, contact: 127.0.0.1:40923]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:23:08,049-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142107.202, now=131106-142308.045, channel=TCPChannel [type: server, contact: 127.0.0.1:40923]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel exception handled
-2013-11-06 14:23:08,049-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 14:23:08,064-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 14:23:08,064-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 14:23:08,065-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:40923] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 14:23:08,065-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:40923] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 14:23:08,065-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:40923]
-2013-11-06 14:23:08,065-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:40923]
-No streams
-2013-11-06 14:23:13,380-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:23:23,245-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 14:23:23,388-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:23:30,980-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 14:23:33,395-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:23:43,403-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:23:53,411-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 14:24:00,564-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 14:24:03,419-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 14:24:07,455-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 14:24:12,873-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:40924] for addition
-2013-11-06 14:24:12,874-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:40924] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:40924]
-2013-11-06 14:24:12,874-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:40924]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fd7
-2013-11-06 14:24:12,908-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fd7
-MetaChannel [context: cps-40924, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:40924]
-2013-11-06 14:24:12,909-0600 INFO MetaChannel MetaChannel [context: cps-40924, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:40924]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 14:24:12,909-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 14:24:12,949-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 14:24:12,949-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 14:24:13,030-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 14:24:13,073-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 29.0) submitting urn:job-0-1383759274570-1383759274571: /bin/echo
-2013-11-06 14:24:13,073-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 29.0) submitting urn:job-0-1383759274570-1383759274571: /bin/echo
-2013-11-06 14:24:13,074-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 14:24:13,074-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-06 14:24:13,082-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7390, JOBSTATUS) unregistering (send)
-2013-11-06 14:24:13,082-0600 INFO RequestHandler Handler(tag: 7390, JOBSTATUS) unregistering (send)
-2013-11-06 14:24:13,084-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 14:24:13,084-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 14:24:13,084-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-06 14:24:13,084-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7391, JOBSTATUS) unregistering (send)
-2013-11-06 14:24:13,084-0600 INFO RequestHandler Handler(tag: 7391, JOBSTATUS) unregistering (send)
-2013-11-06 14:24:13,085-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-Avg stream buf: 0
-2013-11-06 14:24:13,431-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 14:24:23,440-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 14:24:33,449-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:24:37,881-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 14:24:43,458-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 14:24:43,929-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-Avg stream buf: 0
-2013-11-06 14:24:53,467-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 14:25:03,476-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 14:25:13,484-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:25:15,199-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:25:20,402-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 14:25:23,493-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 14:25:33,501-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 14:25:43,510-0600 INFO Multiplexer Avg stream buf: 0
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 14:25:50,251-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:40926] for addition
-2013-11-06 14:25:50,251-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:40926] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:40926]
-2013-11-06 14:25:50,251-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:40926]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fd6
-2013-11-06 14:25:50,291-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fd6
-MetaChannel [context: cps-40926, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:40926]
-2013-11-06 14:25:50,291-0600 INFO MetaChannel MetaChannel [context: cps-40926, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:40926]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 14:25:50,291-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 14:25:50,311-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 14:25:50,311-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 14:25:50,392-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 14:25:50,395-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 30.0) submitting urn:job-0-1383759274573-1383759274574: /bin/echo
-2013-11-06 14:25:50,396-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 30.0) submitting urn:job-0-1383759274573-1383759274574: /bin/echo
-2013-11-06 14:25:50,396-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 14:25:50,396-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-06 14:25:50,404-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7392, JOBSTATUS) unregistering (send)
-2013-11-06 14:25:50,405-0600 INFO RequestHandler Handler(tag: 7392, JOBSTATUS) unregistering (send)
-2013-11-06 14:25:50,407-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 14:25:50,407-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 14:25:50,407-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-06 14:25:50,407-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7393, JOBSTATUS) unregistering (send)
-2013-11-06 14:25:50,407-0600 INFO RequestHandler Handler(tag: 7393, JOBSTATUS) unregistering (send)
-2013-11-06 14:25:50,407-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:25:52,517-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 14:25:53,521-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:25:56,876-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 14:26:03,530-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 14:26:13,539-0600 INFO Multiplexer Avg stream buf: 0
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142413.030, now=131106-142613.873, channel=TCPChannel [type: server, contact: 127.0.0.1:40924]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:26:13,873-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142413.030, now=131106-142613.873, channel=TCPChannel [type: server, contact: 127.0.0.1:40924]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 14:26:13,874-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142413.030, now=131106-142613.873, channel=TCPChannel [type: server, contact: 127.0.0.1:40924]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:26:13,874-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142413.030, now=131106-142613.873, channel=TCPChannel [type: server, contact: 127.0.0.1:40924]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 14:26:13,874-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142413.030, now=131106-142613.873, channel=TCPChannel [type: server, contact: 127.0.0.1:40924]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:26:13,874-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142413.030, now=131106-142613.873, channel=TCPChannel [type: server, contact: 127.0.0.1:40924]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 14:26:13,874-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142413.030, now=131106-142613.873, channel=TCPChannel [type: server, contact: 127.0.0.1:40924]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:26:13,874-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142413.030, now=131106-142613.873, channel=TCPChannel [type: server, contact: 127.0.0.1:40924]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 14:26:13,875-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142413.030, now=131106-142613.873, channel=TCPChannel [type: server, contact: 127.0.0.1:40924]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:26:13,875-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142413.030, now=131106-142613.873, channel=TCPChannel [type: server, contact: 127.0.0.1:40924]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 14:26:13,875-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:40924] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142413.030, now=131106-142613.873, channel=TCPChannel [type: server, contact: 127.0.0.1:40924]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:26:13,875-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:40924] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142413.030, now=131106-142613.873, channel=TCPChannel [type: server, contact: 127.0.0.1:40924]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-null: Channel shut down
-2013-11-06 14:26:13,875-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-40924, boundTo: TCPChannel [type: server, contact: 127.0.0.1:40924]] binding to NullChannel
-2013-11-06 14:26:13,875-0600 INFO MetaChannel MetaChannel [context: cps-40924, boundTo: TCPChannel [type: server, contact: 127.0.0.1:40924]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:40924]
-2013-11-06 14:26:13,875-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:40924]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142413.030, now=131106-142613.873, channel=TCPChannel [type: server, contact: 127.0.0.1:40924]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:26:13,875-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142413.030, now=131106-142613.873, channel=TCPChannel [type: server, contact: 127.0.0.1:40924]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 14:26:13,876-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142413.030, now=131106-142613.873, channel=TCPChannel [type: server, contact: 127.0.0.1:40924]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:26:13,876-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142413.030, now=131106-142613.873, channel=TCPChannel [type: server, contact: 127.0.0.1:40924]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 14:26:13,876-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142413.030, now=131106-142613.873, channel=TCPChannel [type: server, contact: 127.0.0.1:40924]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:26:13,876-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142413.030, now=131106-142613.873, channel=TCPChannel [type: server, contact: 127.0.0.1:40924]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 14:26:13,876-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142413.030, now=131106-142613.873, channel=TCPChannel [type: server, contact: 127.0.0.1:40924]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:26:13,876-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142413.030, now=131106-142613.873, channel=TCPChannel [type: server, contact: 127.0.0.1:40924]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 14:26:13,876-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142413.030, now=131106-142613.873, channel=TCPChannel [type: server, contact: 127.0.0.1:40924]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:26:13,876-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142413.030, now=131106-142613.873, channel=TCPChannel [type: server, contact: 127.0.0.1:40924]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel exception handled
-2013-11-06 14:26:13,877-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 14:26:13,880-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 14:26:13,880-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 14:26:13,881-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:40924] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 14:26:13,881-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:40924] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 14:26:13,881-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:40924]
-2013-11-06 14:26:13,881-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:40924]
-Avg stream buf: 0
-2013-11-06 14:26:23,549-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:26:29,835-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:26:33,350-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 14:26:33,557-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 14:26:43,566-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 14:26:53,575-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 14:27:03,583-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:27:07,153-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:27:09,824-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 14:27:13,592-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 14:27:23,600-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 14:27:33,608-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 14:27:43,617-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:27:44,471-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:27:46,299-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142550.392, now=131106-142751.252, channel=TCPChannel [type: server, contact: 127.0.0.1:40926]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:27:51,252-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142550.392, now=131106-142751.252, channel=TCPChannel [type: server, contact: 127.0.0.1:40926]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 14:27:51,252-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142550.392, now=131106-142751.252, channel=TCPChannel [type: server, contact: 127.0.0.1:40926]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:27:51,252-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142550.392, now=131106-142751.252, channel=TCPChannel [type: server, contact: 127.0.0.1:40926]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 14:27:51,252-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142550.392, now=131106-142751.252, channel=TCPChannel [type: server, contact: 127.0.0.1:40926]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:27:51,252-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142550.392, now=131106-142751.252, channel=TCPChannel [type: server, contact: 127.0.0.1:40926]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 14:27:51,252-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142550.392, now=131106-142751.252, channel=TCPChannel [type: server, contact: 127.0.0.1:40926]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:27:51,252-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142550.392, now=131106-142751.252, channel=TCPChannel [type: server, contact: 127.0.0.1:40926]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 14:27:51,253-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142550.392, now=131106-142751.252, channel=TCPChannel [type: server, contact: 127.0.0.1:40926]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:27:51,253-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142550.392, now=131106-142751.252, channel=TCPChannel [type: server, contact: 127.0.0.1:40926]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 14:27:51,253-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:40926] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142550.392, now=131106-142751.252, channel=TCPChannel [type: server, contact: 127.0.0.1:40926]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:27:51,253-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:40926] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142550.392, now=131106-142751.252, channel=TCPChannel [type: server, contact: 127.0.0.1:40926]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-null: Channel shut down
-2013-11-06 14:27:51,253-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-40926, boundTo: TCPChannel [type: server, contact: 127.0.0.1:40926]] binding to NullChannel
-2013-11-06 14:27:51,253-0600 INFO MetaChannel MetaChannel [context: cps-40926, boundTo: TCPChannel [type: server, contact: 127.0.0.1:40926]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:40926]
-2013-11-06 14:27:51,253-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:40926]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142550.392, now=131106-142751.252, channel=TCPChannel [type: server, contact: 127.0.0.1:40926]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:27:51,254-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142550.392, now=131106-142751.252, channel=TCPChannel [type: server, contact: 127.0.0.1:40926]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 14:27:51,254-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142550.392, now=131106-142751.252, channel=TCPChannel [type: server, contact: 127.0.0.1:40926]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:27:51,254-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142550.392, now=131106-142751.252, channel=TCPChannel [type: server, contact: 127.0.0.1:40926]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 14:27:51,254-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142550.392, now=131106-142751.252, channel=TCPChannel [type: server, contact: 127.0.0.1:40926]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:27:51,254-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142550.392, now=131106-142751.252, channel=TCPChannel [type: server, contact: 127.0.0.1:40926]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 14:27:51,254-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142550.392, now=131106-142751.252, channel=TCPChannel [type: server, contact: 127.0.0.1:40926]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:27:51,254-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142550.392, now=131106-142751.252, channel=TCPChannel [type: server, contact: 127.0.0.1:40926]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 14:27:51,254-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142550.392, now=131106-142751.252, channel=TCPChannel [type: server, contact: 127.0.0.1:40926]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:27:51,254-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-142550.392, now=131106-142751.252, channel=TCPChannel [type: server, contact: 127.0.0.1:40926]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel exception handled
-2013-11-06 14:27:51,255-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 14:27:51,259-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 14:27:51,259-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 14:27:51,259-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:40926] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 14:27:51,259-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:40926] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 14:27:51,259-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:40926]
-2013-11-06 14:27:51,259-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:40926]
-No streams
-2013-11-06 14:27:53,626-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:28:03,634-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:28:13,642-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 3ms
-2013-11-06 14:28:21,791-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 3ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:28:22,773-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 14:28:23,650-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:28:33,659-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:28:43,667-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:28:53,675-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:28:59,107-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 22ms
-2013-11-06 14:28:59,268-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 22ms
-No streams
-2013-11-06 14:29:03,683-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:29:13,692-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:29:23,700-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:29:33,708-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:29:35,721-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 14:29:36,425-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 14:29:43,716-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:29:53,724-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:30:03,732-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:30:12,195-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 14:30:13,740-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:30:13,744-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 14:30:23,748-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:30:33,757-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:30:43,765-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 14:30:48,670-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:30:51,062-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 14:30:53,773-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:31:03,781-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:31:13,789-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:31:23,797-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 4ms
-2013-11-06 14:31:25,146-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 4ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:31:28,380-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 14:31:33,805-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:31:43,813-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:31:53,821-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:32:01,617-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 14:32:03,829-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:32:05,698-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 14:32:13,837-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:32:23,845-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:32:33,853-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:32:38,091-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:32:43,016-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 14:32:43,861-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:32:53,869-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:33:03,876-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:33:13,884-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:33:14,565-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:33:20,334-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 14:33:23,892-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:33:33,900-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:33:43,908-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:33:51,039-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 14:33:53,916-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:33:57,652-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 14:34:03,924-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:34:13,932-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:34:23,940-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:34:27,513-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 14:34:33,948-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:34:34,970-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 14:34:43,956-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:34:53,964-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:35:03,972-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:35:03,988-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:35:12,288-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 14:35:13,980-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:35:23,988-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:35:33,996-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 14:35:40,462-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 14:35:44,004-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:35:49,606-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 14:35:54,012-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:36:04,020-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:36:14,028-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:36:16,935-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 14:36:24,036-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:36:26,924-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 14:36:34,044-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:36:44,052-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:36:53,409-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 14:36:54,060-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:37:04,068-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:37:04,242-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 14:37:14,076-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:37:24,084-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:37:29,883-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 14:37:34,092-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:37:41,560-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 14:37:44,100-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:37:54,108-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:38:04,116-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 14:38:06,358-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 14:38:14,124-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:38:18,878-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 14:38:24,132-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:38:34,140-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:38:42,831-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 14:38:44,148-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:38:54,156-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:38:56,196-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 14:39:04,165-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:39:14,173-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:39:19,305-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 14:39:24,181-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 14:39:33,513-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 14:39:34,190-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:39:44,198-0600 INFO Multiplexer No streams
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 14:39:45,119-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:40929] for addition
-2013-11-06 14:39:45,120-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:40929] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:40929]
-2013-11-06 14:39:45,120-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:40929]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fd5
-2013-11-06 14:39:45,142-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fd5
-MetaChannel [context: cps-40929, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:40929]
-2013-11-06 14:39:45,142-0600 INFO MetaChannel MetaChannel [context: cps-40929, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:40929]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 14:39:45,142-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 14:39:45,203-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 14:39:45,203-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 14:39:45,303-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 14:39:45,348-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 31.0) submitting urn:job-0-1383759274576-1383759274577: /bin/echo
-2013-11-06 14:39:45,348-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 31.0) submitting urn:job-0-1383759274576-1383759274577: /bin/echo
-2013-11-06 14:39:45,349-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 14:39:45,349-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-06 14:39:45,357-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7394, JOBSTATUS) unregistering (send)
-2013-11-06 14:39:45,357-0600 INFO RequestHandler Handler(tag: 7394, JOBSTATUS) unregistering (send)
-2013-11-06 14:39:45,359-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 14:39:45,359-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 14:39:45,359-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-06 14:39:45,360-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7395, JOBSTATUS) unregistering (send)
-2013-11-06 14:39:45,360-0600 INFO RequestHandler Handler(tag: 7395, JOBSTATUS) unregistering (send)
-2013-11-06 14:39:45,360-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-Avg stream buf: 0
-2013-11-06 14:39:54,209-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:39:55,779-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 14:40:04,218-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:40:10,832-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 14:40:14,227-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 14:40:24,236-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:40:32,253-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 14:40:34,245-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 14:40:44,254-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:40:48,150-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 14:40:54,262-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 14:41:04,271-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 14:41:08,728-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-Avg stream buf: 0
-2013-11-06 14:41:14,280-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 14:41:24,289-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:41:25,469-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 14:41:34,297-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 14:41:44,306-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:41:45,201-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-143945.303, now=131106-144146.120, channel=TCPChannel [type: server, contact: 127.0.0.1:40929]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:41:46,120-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-143945.303, now=131106-144146.120, channel=TCPChannel [type: server, contact: 127.0.0.1:40929]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 14:41:46,121-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-143945.303, now=131106-144146.120, channel=TCPChannel [type: server, contact: 127.0.0.1:40929]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:41:46,121-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-143945.303, now=131106-144146.120, channel=TCPChannel [type: server, contact: 127.0.0.1:40929]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 14:41:46,121-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-143945.303, now=131106-144146.120, channel=TCPChannel [type: server, contact: 127.0.0.1:40929]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:41:46,121-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-143945.303, now=131106-144146.120, channel=TCPChannel [type: server, contact: 127.0.0.1:40929]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 14:41:46,121-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-143945.303, now=131106-144146.120, channel=TCPChannel [type: server, contact: 127.0.0.1:40929]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:41:46,121-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-143945.303, now=131106-144146.120, channel=TCPChannel [type: server, contact: 127.0.0.1:40929]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 14:41:46,122-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-143945.303, now=131106-144146.120, channel=TCPChannel [type: server, contact: 127.0.0.1:40929]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:41:46,122-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-143945.303, now=131106-144146.120, channel=TCPChannel [type: server, contact: 127.0.0.1:40929]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 14:41:46,122-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:40929] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-143945.303, now=131106-144146.120, channel=TCPChannel [type: server, contact: 127.0.0.1:40929]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:41:46,122-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:40929] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-143945.303, now=131106-144146.120, channel=TCPChannel [type: server, contact: 127.0.0.1:40929]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-null: Channel shut down
-2013-11-06 14:41:46,122-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-40929, boundTo: TCPChannel [type: server, contact: 127.0.0.1:40929]] binding to NullChannel
-2013-11-06 14:41:46,122-0600 INFO MetaChannel MetaChannel [context: cps-40929, boundTo: TCPChannel [type: server, contact: 127.0.0.1:40929]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:40929]
-2013-11-06 14:41:46,123-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:40929]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-143945.303, now=131106-144146.120, channel=TCPChannel [type: server, contact: 127.0.0.1:40929]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:41:46,123-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-143945.303, now=131106-144146.120, channel=TCPChannel [type: server, contact: 127.0.0.1:40929]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 14:41:46,123-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-143945.303, now=131106-144146.120, channel=TCPChannel [type: server, contact: 127.0.0.1:40929]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:41:46,123-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-143945.303, now=131106-144146.120, channel=TCPChannel [type: server, contact: 127.0.0.1:40929]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 14:41:46,123-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-143945.303, now=131106-144146.120, channel=TCPChannel [type: server, contact: 127.0.0.1:40929]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:41:46,123-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-143945.303, now=131106-144146.120, channel=TCPChannel [type: server, contact: 127.0.0.1:40929]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 14:41:46,123-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-143945.303, now=131106-144146.120, channel=TCPChannel [type: server, contact: 127.0.0.1:40929]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:41:46,123-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-143945.303, now=131106-144146.120, channel=TCPChannel [type: server, contact: 127.0.0.1:40929]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 14:41:46,123-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-143945.303, now=131106-144146.120, channel=TCPChannel [type: server, contact: 127.0.0.1:40929]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:41:46,124-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-143945.303, now=131106-144146.120, channel=TCPChannel [type: server, contact: 127.0.0.1:40929]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel exception handled
-2013-11-06 14:41:46,124-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 14:41:46,131-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 14:41:46,132-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 14:41:46,132-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:40929] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 14:41:46,132-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:40929] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 14:41:46,132-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:40929]
-2013-11-06 14:41:46,132-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:40929]
-No streams
-2013-11-06 14:41:54,315-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:42:02,787-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 14:42:04,323-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:42:14,331-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:42:21,675-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 14:42:24,339-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:42:34,347-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:42:40,105-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 14:42:44,355-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:42:54,363-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:42:58,149-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 14:43:04,371-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:43:14,379-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:43:17,423-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 14:43:24,387-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:43:34,395-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:43:34,623-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 14:43:44,403-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:43:54,412-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:43:54,742-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 14:44:04,420-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:44:11,097-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 14:44:14,428-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:44:24,436-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:44:32,060-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 14:44:34,444-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:44:44,452-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 14:44:47,572-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 14:44:54,461-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:45:04,469-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:45:09,378-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 14:45:14,477-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:45:24,045-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 14:45:24,486-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:45:34,494-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:45:44,502-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:45:46,696-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 14:45:54,510-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:46:00,519-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 14:46:04,518-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:46:14,526-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:46:24,014-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 14:46:24,534-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:46:34,542-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:46:36,993-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 14:46:44,552-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:46:54,561-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 14:47:01,332-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 14:47:04,569-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 14:47:13,468-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 14:47:14,577-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:47:24,586-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:47:34,594-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:47:38,651-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 14:47:44,602-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 14:47:49,941-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 14:47:54,610-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:48:04,619-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:48:14,627-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:48:15,969-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 14:48:24,635-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:48:26,415-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 14:48:34,643-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:48:44,651-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:48:53,287-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 14:48:54,660-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 14:49:02,890-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 14:49:04,668-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:49:14,676-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:49:24,684-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:49:30,605-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 14:49:34,692-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:49:39,363-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 14:49:44,701-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:49:54,709-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:50:04,717-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:50:07,924-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 14:50:14,725-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:50:15,838-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 14:50:24,734-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:50:34,742-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:50:44,750-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:50:45,242-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:50:52,312-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 14:50:54,759-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:51:04,767-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:51:14,775-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:51:22,560-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 14:51:24,783-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 14:51:28,787-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 14:51:34,791-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:51:44,799-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:51:54,807-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:51:59,878-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 14:52:04,815-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:52:05,260-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 14:52:14,823-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:52:24,831-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:52:34,839-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:52:37,196-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 14:52:41,734-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 14:52:44,848-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:52:54,856-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:53:04,864-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 14:53:14,515-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 14:53:14,872-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:53:18,209-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 14:53:24,881-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:53:34,889-0600 INFO Multiplexer No streams
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 14:53:44,192-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:40945] for addition
-2013-11-06 14:53:44,192-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:40945] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:40945]
-2013-11-06 14:53:44,192-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:40945]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fd4
-2013-11-06 14:53:44,216-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fd4
-MetaChannel [context: cps-40945, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:40945]
-2013-11-06 14:53:44,217-0600 INFO MetaChannel MetaChannel [context: cps-40945, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:40945]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 14:53:44,217-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 14:53:44,258-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 14:53:44,258-0600 INFO Sender Sender TCPChannel queue size: 0
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 14:53:44,295-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:40946] for addition
-2013-11-06 14:53:44,295-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:40946] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:40946]
-2013-11-06 14:53:44,296-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:40946]
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7fd3
-2013-11-06 14:53:44,298-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7fd3
-MetaChannel [context: cps-40946, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:40946]
-2013-11-06 14:53:44,298-0600 INFO MetaChannel MetaChannel [context: cps-40946, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:40946]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-06 14:53:44,298-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 14:53:44,339-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-06 14:53:44,339-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-06 14:53:44,349-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 32.0) submitting urn:job-0-1383759274579-1383759274580: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 14:53:44,349-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 32.0) submitting urn:job-0-1383759274579-1383759274580: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 14:53:44,350-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 14:53:44,350-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-06 14:53:44,356-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7396, JOBSTATUS) unregistering (send)
-2013-11-06 14:53:44,356-0600 INFO RequestHandler Handler(tag: 7396, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 14:53:44,384-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:40947] for addition
-2013-11-06 14:53:44,384-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:40947] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:40947]
-2013-11-06 14:53:44,384-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:40947]
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7fd2
-2013-11-06 14:53:44,400-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7fd2
-MetaChannel [context: cps-40947, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:40947]
-2013-11-06 14:53:44,400-0600 INFO MetaChannel MetaChannel [context: cps-40947, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:40947]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-06 14:53:44,400-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 14:53:44,440-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-06 14:53:44,441-0600 INFO RequestHandler Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-06 14:53:44,450-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-crush.mcs.anl.gov:0 (quality: 1.0) submitting urn:job-1-1383759274582-1383759274583: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 14:53:44,450-0600 INFO Cpu crush.mcs.anl.gov:0 (quality: 1.0) submitting urn:job-1-1383759274582-1383759274583: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-LOG: /homes/yadunand/logs/worker-crush.mcs.anl.gov.log
-2013-11-06 14:53:44,458-0600 DEBUG Cpu Active workerid=crush.mcs.anl.gov:000000
-Handler(tag: 33, JOBSTATUS) unregistering (send)
-2013-11-06 14:53:44,459-0600 INFO RequestHandler Handler(tag: 33, JOBSTATUS) unregistering (send)
-org.globus.cog.coaster.ProtocolException: Invalid line:
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler$Helper.scan(SubmitJobHandler.java:312)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler$Helper.read(SubmitJobHandler.java:270)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.read(SubmitJobHandler.java:97)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.requestComplete(SubmitJobHandler.java:61)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:173)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Handler(tag: 42566, SUBMITJOB) sending error: Could not deserialize job description
-org.globus.cog.coaster.ProtocolException: Could not deserialize job description
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.requestComplete(SubmitJobHandler.java:71)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:173)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Caused by: org.globus.cog.coaster.ProtocolException: Invalid line:
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler$Helper.scan(SubmitJobHandler.java:312)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler$Helper.read(SubmitJobHandler.java:270)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.read(SubmitJobHandler.java:97)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.requestComplete(SubmitJobHandler.java:61)
- ... 4 more
-2013-11-06 14:53:44,541-0600 INFO RequestReply Handler(tag: 42566, SUBMITJOB) sending error: Could not deserialize job description
-org.globus.cog.coaster.ProtocolException: Could not deserialize job description
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.requestComplete(SubmitJobHandler.java:71)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:173)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Caused by: org.globus.cog.coaster.ProtocolException: Invalid line:
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler$Helper.scan(SubmitJobHandler.java:312)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler$Helper.read(SubmitJobHandler.java:270)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.read(SubmitJobHandler.java:97)
- at org.globus.cog.abstraction.coaster.service.SubmitJobHandler.requestComplete(SubmitJobHandler.java:61)
- ... 4 more
-Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-06 14:53:44,543-0600 INFO RequestHandler Handler(tag: 42566, SUBMITJOB) unregistering (send)
-Avg stream buf: 0
-2013-11-06 14:53:44,904-0600 INFO Multiplexer Avg stream buf: 0
-2013-11-06 14:53:46,408-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 14:53:46,408-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 14:53:46,408-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 14:53:46,408-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 14:53:46,409-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-Handler(tag: 7397, JOBSTATUS) unregistering (send)
-2013-11-06 14:53:46,408-0600 INFO RequestHandler Handler(tag: 7397, JOBSTATUS) unregistering (send)
-2013-11-06 14:53:46,409-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-Channel config: null -> {keepalive=null}
-2013-11-06 14:53:46,409-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:40945] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 14:53:46,409-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:40945] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-06 14:53:46,409-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-40945, boundTo: TCPChannel [type: server, contact: 127.0.0.1:40945]] binding to NullChannel
-2013-11-06 14:53:46,409-0600 INFO MetaChannel MetaChannel [context: cps-40945, boundTo: TCPChannel [type: server, contact: 127.0.0.1:40945]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:40945]
-2013-11-06 14:53:46,410-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:40945]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 14:53:46,410-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 14:53:46,410-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 14:53:46,410-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 14:53:46,410-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 14:53:46,410-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 14:53:46,410-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 14:53:46,411-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 14:53:46,411-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 14:53:46,411-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 14:53:46,411-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 14:53:46,428-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 14:53:46,428-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 14:53:46,429-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:40945] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 14:53:46,429-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:40945] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 14:53:46,429-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:40945]
-2013-11-06 14:53:46,429-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:40945]
-2013-11-06 14:53:46,851-0600 DEBUG Cpu Completed
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 14:53:46,851-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-crush.mcs.anl.gov:0 jobTerminated
-2013-11-06 14:53:46,851-0600 INFO Cpu crush.mcs.anl.gov:0 jobTerminated
-Channel config: null -> {keepalive=null}
-2013-11-06 14:53:46,852-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-06 14:53:46,852-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-TCPChannel [type: server, contact: 127.0.0.1:40946] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 14:53:46,852-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:40946] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 14:53:46,852-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 34, JOBSTATUS) unregistering (send)
-2013-11-06 14:53:46,852-0600 INFO RequestHandler Handler(tag: 34, JOBSTATUS) unregistering (send)
-2013-11-06 14:53:46,852-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-null: Channel shut down
-2013-11-06 14:53:46,852-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-40946, boundTo: TCPChannel [type: server, contact: 127.0.0.1:40946]] binding to NullChannel
-2013-11-06 14:53:46,852-0600 INFO MetaChannel MetaChannel [context: cps-40946, boundTo: TCPChannel [type: server, contact: 127.0.0.1:40946]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:40946]
-2013-11-06 14:53:46,852-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:40946]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 14:53:46,852-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 14:53:46,853-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 14:53:46,853-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 14:53:46,853-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 14:53:46,853-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 14:53:46,853-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 14:53:46,853-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 14:53:46,854-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 14:53:46,854-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 14:53:46,854-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 14:53:46,870-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 14:53:46,870-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 14:53:46,871-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:40946] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 14:53:46,871-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:40946] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 14:53:46,871-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:40946]
-2013-11-06 14:53:46,871-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:40946]
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:53:51,833-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 14:53:52,664-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:40948] for addition
-2013-11-06 14:53:52,664-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:40948] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:40948]
-2013-11-06 14:53:52,664-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:40948]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fd1
-2013-11-06 14:53:52,688-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fd1
-MetaChannel [context: cps-40948, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:40948]
-2013-11-06 14:53:52,689-0600 INFO MetaChannel MetaChannel [context: cps-40948, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:40948]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 14:53:52,689-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 14:53:52,729-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 14:53:52,768-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:40949] for addition
-2013-11-06 14:53:52,768-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:40949] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:40949]
-2013-11-06 14:53:52,768-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:40949]
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7fd0
-2013-11-06 14:53:52,770-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7fd0
-MetaChannel [context: cps-40949, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:40949]
-2013-11-06 14:53:52,770-0600 INFO MetaChannel MetaChannel [context: cps-40949, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:40949]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-06 14:53:52,770-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-06 14:53:52,811-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 14:53:52,811-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 14:53:52,856-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:40950] for addition
-2013-11-06 14:53:52,856-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:40950] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:40950]
-2013-11-06 14:53:52,856-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:40950]
-2013-11-06 14:53:52,860-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 33.0) submitting urn:job-0-1383759274588-1383759274589: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 14:53:52,860-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 33.0) submitting urn:job-0-1383759274588-1383759274589: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 14:53:52,863-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 14:53:52,863-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7fcf
-2013-11-06 14:53:52,871-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7fcf
-MetaChannel [context: cps-40950, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:40950]
-2013-11-06 14:53:52,872-0600 INFO MetaChannel MetaChannel [context: cps-40950, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:40950]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-06 14:53:52,872-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-06 14:53:52,880-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7398, JOBSTATUS) unregistering (send)
-2013-11-06 14:53:52,880-0600 INFO RequestHandler Handler(tag: 7398, JOBSTATUS) unregistering (send)
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 14:53:52,912-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-06 14:53:52,912-0600 INFO RequestHandler Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-06 14:53:52,913-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-crush.mcs.anl.gov:0 (quality: 2.0) submitting urn:job-1-1383759274591-1383759274592: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 14:53:52,913-0600 INFO Cpu crush.mcs.anl.gov:0 (quality: 2.0) submitting urn:job-1-1383759274591-1383759274592: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 14:53:52,919-0600 DEBUG Cpu Active workerid=crush.mcs.anl.gov:000000
-Handler(tag: 35, JOBSTATUS) unregistering (send)
-2013-11-06 14:53:52,919-0600 INFO RequestHandler Handler(tag: 35, JOBSTATUS) unregistering (send)
-Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-06 14:53:53,013-0600 INFO RequestHandler Handler(tag: 42566, SUBMITJOB) unregistering (send)
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:53:54,683-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:53:54,893-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 14:53:54,893-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 14:53:54,893-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 14:53:54,893-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Channel config: null -> {keepalive=null}
-2013-11-06 14:53:54,893-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-Handler(tag: 7399, JOBSTATUS) unregistering (send)
-2013-11-06 14:53:54,893-0600 INFO RequestHandler Handler(tag: 7399, JOBSTATUS) unregistering (send)
-2013-11-06 14:53:54,893-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-TCPChannel [type: server, contact: 127.0.0.1:40948] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 14:53:54,893-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:40948] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-grind.mcs.anl.gov:0 (quality: 34.0) submitting urn:job-2-1383759274594-1383759274595: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 14:53:54,893-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 34.0) submitting urn:job-2-1383759274594-1383759274595: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-null: Channel shut down
-2013-11-06 14:53:54,894-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-40948, boundTo: TCPChannel [type: server, contact: 127.0.0.1:40948]] binding to NullChannel
-2013-11-06 14:53:54,894-0600 INFO MetaChannel MetaChannel [context: cps-40948, boundTo: TCPChannel [type: server, contact: 127.0.0.1:40948]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:40948]
-2013-11-06 14:53:54,894-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:40948]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 14:53:54,894-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 14:53:54,894-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 14:53:54,894-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 14:53:54,895-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 14:53:54,895-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 14:53:54,895-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 14:53:54,895-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 14:53:54,895-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 14:53:54,895-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 14:53:54,896-0600 INFO ChannelManager Channel exception handled
-Sender TCPChannel queue size: 5
-2013-11-06 14:53:54,896-0600 INFO Sender Sender TCPChannel queue size: 5
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 14:53:54,896-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel config: null -> {keepalive=null}
-2013-11-06 14:53:54,896-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:40950] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 14:53:54,896-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:40950] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-06 14:53:54,896-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-40950, boundTo: TCPChannel [type: server, contact: 127.0.0.1:40950]] binding to NullChannel
-2013-11-06 14:53:54,896-0600 INFO MetaChannel MetaChannel [context: cps-40950, boundTo: TCPChannel [type: server, contact: 127.0.0.1:40950]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:40950]
-2013-11-06 14:53:54,897-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:40950]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 14:53:54,897-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 14:53:54,897-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 14:53:54,897-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 14:53:54,897-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 14:53:54,899-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 14:53:54,899-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 14:53:54,899-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:40950] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 14:53:54,899-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:40950] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 14:53:54,899-0600 INFO ChannelManager Channel already shut down
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 14:53:54,900-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:40950]
-2013-11-06 14:53:54,900-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:40950]
-Channel is closed. Removing.
-2013-11-06 14:53:54,900-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 14:53:54,900-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 14:53:54,900-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:40948] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 14:53:54,900-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:40948] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 14:53:54,901-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:40948]
-2013-11-06 14:53:54,901-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:40948]
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 14:53:54,901-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 14:53:54,902-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7400, JOBSTATUS) unregistering (send)
-2013-11-06 14:53:54,902-0600 INFO RequestHandler Handler(tag: 7400, JOBSTATUS) unregistering (send)
-Avg stream buf: 0
-2013-11-06 14:53:54,921-0600 INFO Multiplexer Avg stream buf: 0
-2013-11-06 14:53:54,932-0600 DEBUG Cpu Completed
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 14:53:54,932-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-crush.mcs.anl.gov:0 jobTerminated
-2013-11-06 14:53:54,932-0600 INFO Cpu crush.mcs.anl.gov:0 jobTerminated
-Channel config: null -> {keepalive=null}
-2013-11-06 14:53:54,933-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-06 14:53:54,933-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-TCPChannel [type: server, contact: 127.0.0.1:40949] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 14:53:54,933-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:40949] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 36, JOBSTATUS) unregistering (send)
-2013-11-06 14:53:54,933-0600 INFO RequestHandler Handler(tag: 36, JOBSTATUS) unregistering (send)
-2013-11-06 14:53:54,933-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 14:53:54,934-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-null: Channel shut down
-2013-11-06 14:53:54,934-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-40949, boundTo: TCPChannel [type: server, contact: 127.0.0.1:40949]] binding to NullChannel
-2013-11-06 14:53:54,934-0600 INFO MetaChannel MetaChannel [context: cps-40949, boundTo: TCPChannel [type: server, contact: 127.0.0.1:40949]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:40949]
-2013-11-06 14:53:54,934-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:40949]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 14:53:54,934-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 14:53:54,934-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 14:53:54,934-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 14:53:54,934-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 14:53:54,934-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 14:53:54,935-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 14:53:54,935-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 14:53:54,935-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 14:53:54,935-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 14:53:54,935-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 14:53:54,941-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 14:53:54,941-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 14:53:54,941-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:40949] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 14:53:54,941-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:40949] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 14:53:54,941-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:40949]
-2013-11-06 14:53:54,942-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:40949]
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 14:53:56,912-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 14:53:56,912-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 14:53:56,912-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 14:53:56,912-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-06 14:53:56,912-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7401, JOBSTATUS) unregistering (send)
-2013-11-06 14:53:56,912-0600 INFO RequestHandler Handler(tag: 7401, JOBSTATUS) unregistering (send)
-2013-11-06 14:53:56,912-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-Avg stream buf: 0
-2013-11-06 14:54:04,930-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 14:54:14,939-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 14:54:24,948-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:54:29,150-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:54:31,157-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 14:54:34,957-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 14:54:44,966-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 14:54:54,974-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 14:55:04,983-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:55:06,468-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:55:07,631-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 14:55:14,992-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 14:55:25,000-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-06 14:55:35,009-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 4ms
-2013-11-06 14:55:43,789-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 4ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:55:44,105-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-06 14:55:45,018-0600 INFO Multiplexer Avg stream buf: 0
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145344.339, now=131106-145545.192, channel=TCPChannel [type: server, contact: 127.0.0.1:40945]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:55:45,192-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145344.339, now=131106-145545.192, channel=TCPChannel [type: server, contact: 127.0.0.1:40945]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 14:55:45,192-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145344.339, now=131106-145545.192, channel=TCPChannel [type: server, contact: 127.0.0.1:40945]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:55:45,192-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145344.339, now=131106-145545.192, channel=TCPChannel [type: server, contact: 127.0.0.1:40945]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 14:55:45,192-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145344.339, now=131106-145545.192, channel=TCPChannel [type: server, contact: 127.0.0.1:40945]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:55:45,192-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145344.339, now=131106-145545.192, channel=TCPChannel [type: server, contact: 127.0.0.1:40945]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 14:55:45,192-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145344.339, now=131106-145545.192, channel=TCPChannel [type: server, contact: 127.0.0.1:40945]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:55:45,193-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145344.339, now=131106-145545.192, channel=TCPChannel [type: server, contact: 127.0.0.1:40945]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 14:55:45,193-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145344.339, now=131106-145545.192, channel=TCPChannel [type: server, contact: 127.0.0.1:40945]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:55:45,193-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145344.339, now=131106-145545.192, channel=TCPChannel [type: server, contact: 127.0.0.1:40945]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 14:55:45,193-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:40945] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145344.339, now=131106-145545.192, channel=TCPChannel [type: server, contact: 127.0.0.1:40945]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:55:45,193-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:40945] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145344.339, now=131106-145545.192, channel=TCPChannel [type: server, contact: 127.0.0.1:40945]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 14:55:45,193-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145344.440, now=131106-145545.295, channel=TCPChannel [type: server, contact: 127.0.0.1:40946]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:55:45,295-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145344.440, now=131106-145545.295, channel=TCPChannel [type: server, contact: 127.0.0.1:40946]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 14:55:45,296-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145344.440, now=131106-145545.295, channel=TCPChannel [type: server, contact: 127.0.0.1:40946]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:55:45,296-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145344.440, now=131106-145545.295, channel=TCPChannel [type: server, contact: 127.0.0.1:40946]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 14:55:45,296-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145344.440, now=131106-145545.295, channel=TCPChannel [type: server, contact: 127.0.0.1:40946]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:55:45,296-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145344.440, now=131106-145545.295, channel=TCPChannel [type: server, contact: 127.0.0.1:40946]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 14:55:45,296-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145344.440, now=131106-145545.295, channel=TCPChannel [type: server, contact: 127.0.0.1:40946]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:55:45,296-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145344.440, now=131106-145545.295, channel=TCPChannel [type: server, contact: 127.0.0.1:40946]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 14:55:45,296-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145344.440, now=131106-145545.295, channel=TCPChannel [type: server, contact: 127.0.0.1:40946]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:55:45,296-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145344.440, now=131106-145545.295, channel=TCPChannel [type: server, contact: 127.0.0.1:40946]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 14:55:45,297-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:40946] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145344.440, now=131106-145545.295, channel=TCPChannel [type: server, contact: 127.0.0.1:40946]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:55:45,297-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:40946] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145344.440, now=131106-145545.295, channel=TCPChannel [type: server, contact: 127.0.0.1:40946]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 14:55:45,297-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145344.541, now=131106-145545.384, channel=TCPChannel [type: server, contact: 127.0.0.1:40947]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:55:45,384-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145344.541, now=131106-145545.384, channel=TCPChannel [type: server, contact: 127.0.0.1:40947]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 14:55:45,384-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:40947] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145344.541, now=131106-145545.384, channel=TCPChannel [type: server, contact: 127.0.0.1:40947]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:55:45,384-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:40947] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145344.541, now=131106-145545.384, channel=TCPChannel [type: server, contact: 127.0.0.1:40947]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-null: Channel shut down
-2013-11-06 14:55:45,384-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-40947, boundTo: TCPChannel [type: server, contact: 127.0.0.1:40947]] binding to NullChannel
-2013-11-06 14:55:45,385-0600 INFO MetaChannel MetaChannel [context: cps-40947, boundTo: TCPChannel [type: server, contact: 127.0.0.1:40947]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:40947]
-2013-11-06 14:55:45,385-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:40947]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145344.541, now=131106-145545.384, channel=TCPChannel [type: server, contact: 127.0.0.1:40947]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:55:45,385-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145344.541, now=131106-145545.384, channel=TCPChannel [type: server, contact: 127.0.0.1:40947]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel exception handled
-2013-11-06 14:55:45,385-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 14:55:45,399-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 14:55:45,399-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 14:55:45,400-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:40947] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 14:55:45,400-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:40947] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 14:55:45,400-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:40947]
-2013-11-06 14:55:45,400-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:40947]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145352.811, now=131106-145553.664, channel=TCPChannel [type: server, contact: 127.0.0.1:40948]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:55:53,664-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145352.811, now=131106-145553.664, channel=TCPChannel [type: server, contact: 127.0.0.1:40948]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 14:55:53,665-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145352.811, now=131106-145553.664, channel=TCPChannel [type: server, contact: 127.0.0.1:40948]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:55:53,665-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145352.811, now=131106-145553.664, channel=TCPChannel [type: server, contact: 127.0.0.1:40948]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 14:55:53,665-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145352.811, now=131106-145553.664, channel=TCPChannel [type: server, contact: 127.0.0.1:40948]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:55:53,665-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145352.811, now=131106-145553.664, channel=TCPChannel [type: server, contact: 127.0.0.1:40948]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 14:55:53,665-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145352.811, now=131106-145553.664, channel=TCPChannel [type: server, contact: 127.0.0.1:40948]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:55:53,665-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145352.811, now=131106-145553.664, channel=TCPChannel [type: server, contact: 127.0.0.1:40948]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 14:55:53,665-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145352.811, now=131106-145553.664, channel=TCPChannel [type: server, contact: 127.0.0.1:40948]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:55:53,665-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145352.811, now=131106-145553.664, channel=TCPChannel [type: server, contact: 127.0.0.1:40948]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 14:55:53,666-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:40948] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145352.811, now=131106-145553.664, channel=TCPChannel [type: server, contact: 127.0.0.1:40948]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:55:53,666-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:40948] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145352.811, now=131106-145553.664, channel=TCPChannel [type: server, contact: 127.0.0.1:40948]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 14:55:53,666-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145352.912, now=131106-145553.768, channel=TCPChannel [type: server, contact: 127.0.0.1:40949]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:55:53,768-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145352.912, now=131106-145553.768, channel=TCPChannel [type: server, contact: 127.0.0.1:40949]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 14:55:53,768-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145352.912, now=131106-145553.768, channel=TCPChannel [type: server, contact: 127.0.0.1:40949]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:55:53,768-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145352.912, now=131106-145553.768, channel=TCPChannel [type: server, contact: 127.0.0.1:40949]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 14:55:53,768-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145352.912, now=131106-145553.768, channel=TCPChannel [type: server, contact: 127.0.0.1:40949]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:55:53,769-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145352.912, now=131106-145553.768, channel=TCPChannel [type: server, contact: 127.0.0.1:40949]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 14:55:53,769-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145352.912, now=131106-145553.768, channel=TCPChannel [type: server, contact: 127.0.0.1:40949]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:55:53,769-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145352.912, now=131106-145553.768, channel=TCPChannel [type: server, contact: 127.0.0.1:40949]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 14:55:53,769-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145352.912, now=131106-145553.768, channel=TCPChannel [type: server, contact: 127.0.0.1:40949]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:55:53,769-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145352.912, now=131106-145553.768, channel=TCPChannel [type: server, contact: 127.0.0.1:40949]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 14:55:53,769-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:40949] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145352.912, now=131106-145553.768, channel=TCPChannel [type: server, contact: 127.0.0.1:40949]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:55:53,769-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:40949] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145352.912, now=131106-145553.768, channel=TCPChannel [type: server, contact: 127.0.0.1:40949]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 14:55:53,769-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145353.013, now=131106-145553.856, channel=TCPChannel [type: server, contact: 127.0.0.1:40950]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:55:53,856-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145353.013, now=131106-145553.856, channel=TCPChannel [type: server, contact: 127.0.0.1:40950]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 14:55:53,856-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145353.013, now=131106-145553.856, channel=TCPChannel [type: server, contact: 127.0.0.1:40950]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:55:53,856-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145353.013, now=131106-145553.856, channel=TCPChannel [type: server, contact: 127.0.0.1:40950]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 14:55:53,856-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:40950] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145353.013, now=131106-145553.856, channel=TCPChannel [type: server, contact: 127.0.0.1:40950]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 14:55:53,856-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:40950] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-145353.013, now=131106-145553.856, channel=TCPChannel [type: server, contact: 127.0.0.1:40950]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 14:55:53,856-0600 INFO ChannelManager Channel already shut down
-No streams
-2013-11-06 14:55:55,027-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:56:05,035-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:56:15,044-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:56:20,580-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:56:21,104-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 14:56:25,052-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:56:35,060-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:56:45,068-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:56:55,077-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:56:57,054-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:56:58,422-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 14:57:05,085-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:57:15,093-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:57:25,101-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:57:33,528-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 14:57:35,109-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:57:35,740-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 14:57:45,117-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:57:55,126-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:58:05,134-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:58:10,002-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:58:13,058-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 14:58:15,142-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:58:25,150-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:58:35,159-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:58:45,167-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:58:46,476-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 14:58:50,376-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 14:58:55,175-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:59:05,183-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:59:15,192-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:59:22,950-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 14:59:25,200-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:59:27,694-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 14:59:35,208-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:59:45,217-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 14:59:55,225-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 14:59:59,424-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 15:00:05,012-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 15:00:05,233-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:00:15,241-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:00:25,249-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:00:35,257-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 17ms
-2013-11-06 15:00:35,914-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 17ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:00:42,330-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:00:45,265-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:00:55,274-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:01:05,282-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:01:12,372-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:01:15,290-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 15:01:19,648-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 15:01:25,298-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:01:35,307-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:01:45,315-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:01:48,846-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:01:55,323-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:01:56,967-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:02:05,331-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:02:15,340-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:02:25,321-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:02:25,348-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:02:34,285-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:02:35,356-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:02:45,365-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:02:55,373-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:03:01,795-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:03:05,381-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 15:03:11,603-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 15:03:15,389-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:03:25,397-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:03:35,405-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 15:03:38,269-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 15:03:45,413-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:03:48,921-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:03:55,422-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:04:05,430-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:04:14,743-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:04:15,438-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:04:25,446-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 15:04:26,239-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 15:04:35,455-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:04:45,463-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:04:51,217-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:04:55,471-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:05:03,557-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:05:05,479-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:05:15,487-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:05:25,496-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:05:27,691-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:05:35,504-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 15:05:40,875-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 15:05:45,512-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:05:55,520-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 15:06:04,165-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 15:06:05,528-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:06:15,536-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:06:18,193-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:06:25,545-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:06:35,553-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:06:40,639-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:06:45,561-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:06:55,511-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:06:55,569-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:07:05,577-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:07:15,585-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 15:07:17,114-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 15:07:25,593-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:07:32,829-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:07:35,601-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:07:45,609-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:07:53,587-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:07:55,617-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:08:05,625-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:08:10,147-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:08:15,633-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:08:25,641-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:08:30,061-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:08:35,649-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:08:45,657-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:08:47,465-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:08:55,665-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:09:05,673-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 15:09:06,537-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 15:09:15,681-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:09:24,783-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:09:25,690-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:09:35,698-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 15:09:43,009-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 15:09:45,706-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:09:55,714-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:10:02,101-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:10:05,722-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:10:15,730-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:10:19,484-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:10:25,738-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:10:35,747-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:10:39,419-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:10:45,755-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:10:55,763-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 10ms
-2013-11-06 15:10:55,966-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 10ms
-No streams
-2013-11-06 15:11:05,772-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:11:15,780-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:11:16,737-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:11:25,788-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-2013-11-06 15:11:32,434-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-No streams
-2013-11-06 15:11:35,796-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:11:45,805-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:11:54,055-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:11:55,814-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:12:05,822-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:12:08,906-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:12:15,833-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:12:25,842-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:12:31,374-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:12:35,853-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 15:12:45,381-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 15:12:45,863-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:12:55,872-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:13:05,883-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:13:08,692-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:13:15,900-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 15:13:21,855-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 15:13:25,914-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:13:35,932-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:13:45,949-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 15:13:46,011-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 15:13:55,962-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:13:58,329-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:14:05,972-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:14:15,981-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:14:23,329-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:14:25,990-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 15:14:34,805-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 15:14:35,999-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:14:46,008-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:14:56,017-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:15:00,647-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:15:06,026-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:15:11,277-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:15:16,036-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:15:26,045-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:15:36,054-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:15:37,965-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:15:46,063-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:15:47,751-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:15:56,072-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:16:06,082-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 15:16:15,284-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 15:16:16,091-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-06 15:16:24,226-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-No streams
-2013-11-06 15:16:26,101-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:16:36,111-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:16:46,120-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:16:52,601-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:16:56,128-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:17:00,699-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:17:06,136-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:17:16,144-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:17:26,152-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 15:17:29,920-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 15:17:36,161-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 15:17:37,174-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 15:17:46,169-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:17:56,177-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:18:06,185-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:18:07,237-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:18:13,647-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:18:16,194-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:18:26,201-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:18:36,209-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 15:18:44,555-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 15:18:46,217-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 15:18:50,121-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 15:18:56,225-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:19:06,233-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:19:16,240-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:19:21,874-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:19:26,248-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 15:19:26,595-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 15:19:36,256-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:19:46,264-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:19:56,272-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:19:59,192-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 15:20:03,069-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 15:20:06,280-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:20:16,287-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:20:26,295-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:20:36,303-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 15:20:36,510-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:20:39,543-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:20:46,311-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:20:56,319-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:21:06,326-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:21:13,828-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 15:21:16,018-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 15:21:16,334-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:21:26,342-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:21:36,350-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:21:46,358-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 15:21:51,147-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:21:52,491-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:21:56,365-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:22:06,373-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:22:16,381-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:22:26,389-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:22:28,464-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 15:22:28,965-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 15:22:36,397-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:22:46,405-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:22:56,413-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 15:23:05,441-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:23:05,782-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:23:06,421-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:23:16,428-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:23:26,436-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:23:36,444-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:23:41,914-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:23:43,100-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:23:46,452-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:23:56,460-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:24:06,468-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:24:16,476-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 15:24:18,389-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:24:20,418-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:24:26,483-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:24:36,491-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:24:46,499-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:24:54,862-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:24:56,507-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 15:24:57,736-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 15:25:06,515-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:25:16,523-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:25:26,530-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 15:25:31,337-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:25:35,054-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:25:36,538-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:25:46,546-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:25:56,554-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:26:06,562-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:26:07,810-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 15:26:12,373-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 15:26:16,570-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:26:26,578-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:26:36,585-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 15:26:44,285-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 15:26:46,593-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:26:49,690-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:26:56,601-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:27:06,610-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:27:16,618-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:27:20,759-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:27:26,626-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:27:27,008-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:27:36,634-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:27:46,643-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:27:56,652-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 15:27:57,233-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:28:04,327-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:28:06,660-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:28:16,669-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:28:26,677-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 15:28:33,707-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 15:28:36,686-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:28:41,644-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:28:46,697-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:28:56,709-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:29:06,720-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:29:10,180-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:29:16,731-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 15:29:18,963-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 15:29:26,742-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:29:36,753-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-06 15:29:46,655-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-No streams
-2013-11-06 15:29:46,762-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:29:56,281-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:29:56,779-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:30:06,784-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:30:16,789-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-06 15:30:23,129-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-No streams
-2013-11-06 15:30:26,791-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:30:33,599-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:30:36,801-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:30:46,811-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:30:56,821-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 15:30:59,603-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 15:31:06,833-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 15:31:10,918-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 15:31:16,845-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:31:26,855-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 15:31:36,077-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 15:31:36,864-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:31:46,878-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 15:31:48,236-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 15:31:56,887-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:32:06,897-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:32:12,551-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:32:16,907-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:32:25,553-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:32:26,917-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:32:36,925-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:32:46,933-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 15:32:49,025-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 15:32:56,941-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 15:33:02,871-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 15:33:06,949-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:33:16,959-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 15:33:25,499-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 15:33:26,968-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:33:36,982-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 15:33:40,190-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 15:33:46,992-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:33:57,001-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:34:01,973-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:34:07,011-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:34:17,019-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:34:17,507-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:34:27,026-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:34:37,035-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:34:38,448-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:34:47,044-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:34:54,825-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:34:57,052-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:35:07,060-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 15:35:14,923-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 15:35:17,069-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:35:27,077-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:35:32,143-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:35:37,084-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:35:47,092-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 15:35:51,396-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 15:35:57,099-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:36:07,107-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 15:36:09,462-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 15:36:17,116-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:36:27,125-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:36:27,871-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:36:37,135-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 15:36:46,779-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 15:36:47,145-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:36:57,156-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:37:04,344-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:37:07,166-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:37:17,176-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 15:37:24,098-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 15:37:27,186-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:37:37,194-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:37:40,819-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:37:47,205-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:37:57,214-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:38:01,415-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:38:07,224-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:38:17,233-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:38:17,293-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:38:27,243-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:38:37,252-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 15:38:38,734-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 15:38:47,262-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 15:38:53,768-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 15:38:57,271-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:39:07,280-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:39:16,051-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:39:17,290-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:39:27,300-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 15:39:30,242-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 15:39:37,309-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:39:47,319-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 15:39:53,370-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 15:39:57,327-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:40:06,715-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:40:07,339-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:40:17,349-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:40:27,360-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:40:30,687-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:40:37,369-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:40:43,189-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:40:47,379-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:40:57,388-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:41:07,396-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 15:41:08,005-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 15:41:17,404-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:41:19,663-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:41:27,411-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:41:37,419-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:41:45,323-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:41:47,427-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:41:56,137-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:41:57,434-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:42:07,442-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:42:17,450-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 15:42:22,641-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 15:42:27,458-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:42:32,611-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:42:37,466-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:42:47,474-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:42:57,482-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:42:59,959-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:43:07,490-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:43:09,085-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:43:17,498-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:43:27,506-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 15:43:37,277-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 15:43:37,514-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 15:43:45,560-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 15:43:47,522-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:43:57,530-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:44:07,538-0600 INFO Multiplexer No streams
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 15:44:14,283-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41092] for addition
-2013-11-06 15:44:14,284-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41092] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41092]
-2013-11-06 15:44:14,284-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41092]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fce
-2013-11-06 15:44:14,318-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fce
-MetaChannel [context: cps-41092, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41092]
-2013-11-06 15:44:14,318-0600 INFO MetaChannel MetaChannel [context: cps-41092, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41092]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 15:44:14,318-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 15:44:14,379-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 15:44:14,379-0600 INFO Sender Sender TCPChannel queue size: 0
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 15:44:14,420-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41093] for addition
-2013-11-06 15:44:14,420-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41093] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41093]
-2013-11-06 15:44:14,420-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41093]
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7fcd
-2013-11-06 15:44:14,439-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7fcd
-MetaChannel [context: cps-41093, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41093]
-2013-11-06 15:44:14,439-0600 INFO MetaChannel MetaChannel [context: cps-41093, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41093]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-06 15:44:14,439-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 15:44:14,460-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-06 15:44:14,480-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-06 15:44:14,488-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 35.0) submitting urn:job-0-1383759274597-1383759274598: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 15:44:14,488-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 35.0) submitting urn:job-0-1383759274597-1383759274598: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 15:44:14,488-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 15:44:14,488-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-06 15:44:14,494-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7402, JOBSTATUS) unregistering (send)
-2013-11-06 15:44:14,494-0600 INFO RequestHandler Handler(tag: 7402, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 15:44:14,519-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41094] for addition
-2013-11-06 15:44:14,520-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41094] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41094]
-2013-11-06 15:44:14,520-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41094]
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7fcc
-2013-11-06 15:44:14,521-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7fcc
-MetaChannel [context: cps-41094, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41094]
-2013-11-06 15:44:14,521-0600 INFO MetaChannel MetaChannel [context: cps-41094, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41094]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-06 15:44:14,521-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 15:44:14,561-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-06 15:44:14,562-0600 INFO RequestHandler Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-06 15:44:14,589-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-crush.mcs.anl.gov:0 (quality: 3.0) submitting urn:job-1-1383759274600-1383759274601: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 15:44:14,589-0600 INFO Cpu crush.mcs.anl.gov:0 (quality: 3.0) submitting urn:job-1-1383759274600-1383759274601: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:44:14,595-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:44:14,596-0600 DEBUG Cpu Active workerid=crush.mcs.anl.gov:000000
-Handler(tag: 37, JOBSTATUS) unregistering (send)
-2013-11-06 15:44:14,596-0600 INFO RequestHandler Handler(tag: 37, JOBSTATUS) unregistering (send)
-Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-06 15:44:14,662-0600 INFO RequestHandler Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-06 15:44:16,514-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 15:44:16,515-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:44:16,515-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:44:16,515-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Channel config: null -> {keepalive=null}
-2013-11-06 15:44:16,515-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-Handler(tag: 7403, JOBSTATUS) unregistering (send)
-2013-11-06 15:44:16,515-0600 INFO RequestHandler Handler(tag: 7403, JOBSTATUS) unregistering (send)
-2013-11-06 15:44:16,515-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-TCPChannel [type: server, contact: 127.0.0.1:41092] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:44:16,515-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41092] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-06 15:44:16,516-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-41092, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41092]] binding to NullChannel
-2013-11-06 15:44:16,516-0600 INFO MetaChannel MetaChannel [context: cps-41092, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41092]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41092]
-2013-11-06 15:44:16,516-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41092]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:44:16,516-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 15:44:16,516-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:44:16,517-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 15:44:16,517-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:44:16,517-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 15:44:16,517-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:44:16,517-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 15:44:16,517-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:44:16,518-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 15:44:16,518-0600 INFO ChannelManager Channel exception handled
-grind.mcs.anl.gov:0 (quality: 36.0) submitting urn:job-2-1383759274603-1383759274604: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 15:44:16,518-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 36.0) submitting urn:job-2-1383759274603-1383759274604: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:44:16,518-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel config: null -> {keepalive=null}
-2013-11-06 15:44:16,519-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41094] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:44:16,519-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41094] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-06 15:44:16,519-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-41094, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41094]] binding to NullChannel
-2013-11-06 15:44:16,519-0600 INFO MetaChannel MetaChannel [context: cps-41094, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41094]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41094]
-2013-11-06 15:44:16,519-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41094]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:44:16,519-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 15:44:16,519-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:44:16,519-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 15:44:16,520-0600 INFO ChannelManager Channel exception handled
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 15:44:16,522-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 15:44:16,524-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 15:44:16,524-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7404, JOBSTATUS) unregistering (send)
-2013-11-06 15:44:16,525-0600 INFO RequestHandler Handler(tag: 7404, JOBSTATUS) unregistering (send)
-Channel is closed. Removing.
-2013-11-06 15:44:16,528-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 15:44:16,528-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 15:44:16,528-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41092] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 15:44:16,528-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41092] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 15:44:16,529-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41092]
-2013-11-06 15:44:16,529-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41092]
-Channel is closed. Removing.
-2013-11-06 15:44:16,529-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 15:44:16,529-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 15:44:16,529-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41094] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 15:44:16,529-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41094] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 15:44:16,529-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41094]
-2013-11-06 15:44:16,529-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41094]
-2013-11-06 15:44:16,734-0600 DEBUG Cpu Completed
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:44:16,734-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-crush.mcs.anl.gov:0 jobTerminated
-2013-11-06 15:44:16,734-0600 INFO Cpu crush.mcs.anl.gov:0 jobTerminated
-Channel config: null -> {keepalive=null}
-2013-11-06 15:44:16,735-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-06 15:44:16,735-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-TCPChannel [type: server, contact: 127.0.0.1:41093] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:44:16,735-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41093] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:44:16,735-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-Handler(tag: 38, JOBSTATUS) unregistering (send)
-2013-11-06 15:44:16,735-0600 INFO RequestHandler Handler(tag: 38, JOBSTATUS) unregistering (send)
-null: Channel shut down
-2013-11-06 15:44:16,735-0600 INFO AbstractTCPChannel null: Channel shut down
-2013-11-06 15:44:16,735-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-MetaChannel [context: cps-41093, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41093]] binding to NullChannel
-2013-11-06 15:44:16,736-0600 INFO MetaChannel MetaChannel [context: cps-41093, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41093]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41093]
-2013-11-06 15:44:16,736-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41093]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:44:16,736-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 15:44:16,736-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:44:16,736-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 15:44:16,736-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:44:16,736-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 15:44:16,737-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:44:16,737-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 15:44:16,737-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:44:16,737-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 15:44:16,737-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 15:44:16,750-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 15:44:16,750-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 15:44:16,750-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41093] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 15:44:16,750-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41093] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 15:44:16,750-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41093]
-2013-11-06 15:44:16,751-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41093]
-No streams
-2013-11-06 15:44:17,553-0600 INFO Multiplexer No streams
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 15:44:18,536-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 15:44:18,536-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 15:44:18,536-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 15:44:18,536-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-06 15:44:18,536-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7405, JOBSTATUS) unregistering (send)
-2013-11-06 15:44:18,536-0600 INFO RequestHandler Handler(tag: 7405, JOBSTATUS) unregistering (send)
-2013-11-06 15:44:18,537-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:44:22,034-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:44:27,561-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:44:37,569-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:44:47,577-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 15:44:51,914-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 15:44:57,585-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 15:44:58,508-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 15:45:07,593-0600 INFO Multiplexer No streams
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 15:45:13,348-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41095] for addition
-2013-11-06 15:45:13,348-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41095] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41095]
-2013-11-06 15:45:13,348-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41095]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fcb
-2013-11-06 15:45:13,369-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fcb
-MetaChannel [context: cps-41095, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41095]
-2013-11-06 15:45:13,370-0600 INFO MetaChannel MetaChannel [context: cps-41095, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41095]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 15:45:13,370-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 15:45:13,430-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 15:45:13,430-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 15:45:13,491-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:45:13,491-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel config: null -> {keepalive=null}
-2013-11-06 15:45:13,492-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41095] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:45:13,492-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41095] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-06 15:45:13,492-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-41095, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41095]] binding to NullChannel
-2013-11-06 15:45:13,492-0600 INFO MetaChannel MetaChannel [context: cps-41095, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41095]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41095]
-2013-11-06 15:45:13,492-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41095]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:45:13,492-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 15:45:13,493-0600 INFO ChannelManager Channel exception handled
-2013-11-06 15:45:13,500-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 37.0) submitting urn:job-0-1383759274606-1383759274607: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 15:45:13,500-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 37.0) submitting urn:job-0-1383759274606-1383759274607: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.submit(Cpu.java:281)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.launchSequential(Cpu.java:244)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.launch(Cpu.java:229)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.pull(Cpu.java:203)
- at org.globus.cog.abstraction.coaster.service.job.manager.PullThread.run(PullThread.java:131)
-2013-11-06 15:45:13,500-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.submit(Cpu.java:281)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.launchSequential(Cpu.java:244)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.launch(Cpu.java:229)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.pull(Cpu.java:203)
- at org.globus.cog.abstraction.coaster.service.job.manager.PullThread.run(PullThread.java:131)
-2013-11-06 15:45:13,500-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 15:45:13,500-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 15:45:13,505-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 15:45:13,507-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 15:45:13,507-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7406, JOBSTATUS) unregistering (send)
-2013-11-06 15:45:13,507-0600 INFO RequestHandler Handler(tag: 7406, JOBSTATUS) unregistering (send)
-Channel is closed. Removing.
-2013-11-06 15:45:13,511-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 15:45:13,512-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 15:45:13,512-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41095] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 15:45:13,512-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41095] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 15:45:13,512-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41095]
-2013-11-06 15:45:13,512-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41095]
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 15:45:15,523-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 15:45:15,524-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 15:45:15,524-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 15:45:15,524-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-06 15:45:15,524-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7407, JOBSTATUS) unregistering (send)
-2013-11-06 15:45:15,524-0600 INFO RequestHandler Handler(tag: 7407, JOBSTATUS) unregistering (send)
-2013-11-06 15:45:15,524-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-No streams
-2013-11-06 15:45:17,604-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:45:27,611-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 15:45:29,232-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 15:45:34,982-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 15:45:37,620-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:45:47,627-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:45:57,635-0600 INFO Multiplexer No streams
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 15:45:58,792-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41096] for addition
-2013-11-06 15:45:58,792-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41096] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41096]
-2013-11-06 15:45:58,792-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41096]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fca
-2013-11-06 15:45:58,819-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fca
-MetaChannel [context: cps-41096, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41096]
-2013-11-06 15:45:58,819-0600 INFO MetaChannel MetaChannel [context: cps-41096, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41096]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 15:45:58,819-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 15:45:58,860-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 15:45:58,860-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 15:45:58,900-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:45:58,900-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel config: null -> {keepalive=null}
-2013-11-06 15:45:58,901-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41096] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:45:58,901-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41096] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-06 15:45:58,901-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-41096, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41096]] binding to NullChannel
-2013-11-06 15:45:58,901-0600 INFO MetaChannel MetaChannel [context: cps-41096, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41096]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41096]
-2013-11-06 15:45:58,901-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41096]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:45:58,902-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 15:45:58,902-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 15:45:58,920-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 15:45:58,921-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 15:45:58,921-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41096] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 15:45:58,921-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41096] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 15:45:58,921-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41096]
-2013-11-06 15:45:58,921-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41096]
-2013-11-06 15:45:58,923-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 38.0) submitting urn:job-0-1383759274609-1383759274610: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 15:45:58,923-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 38.0) submitting urn:job-0-1383759274609-1383759274610: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.submit(Cpu.java:281)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.launchSequential(Cpu.java:244)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.launch(Cpu.java:229)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.pull(Cpu.java:203)
- at org.globus.cog.abstraction.coaster.service.job.manager.PullThread.run(PullThread.java:131)
-2013-11-06 15:45:58,923-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.submit(Cpu.java:281)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.launchSequential(Cpu.java:244)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.launch(Cpu.java:229)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.pull(Cpu.java:203)
- at org.globus.cog.abstraction.coaster.service.job.manager.PullThread.run(PullThread.java:131)
-2013-11-06 15:45:58,924-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 15:45:58,924-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 15:45:58,930-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 15:45:58,931-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 15:45:58,932-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7408, JOBSTATUS) unregistering (send)
-2013-11-06 15:45:58,932-0600 INFO RequestHandler Handler(tag: 7408, JOBSTATUS) unregistering (send)
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 15:46:00,950-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 15:46:00,950-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 15:46:00,950-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 15:46:00,950-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-06 15:46:00,950-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7409, JOBSTATUS) unregistering (send)
-2013-11-06 15:46:00,950-0600 INFO RequestHandler Handler(tag: 7409, JOBSTATUS) unregistering (send)
-2013-11-06 15:46:00,950-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:46:06,549-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:46:07,646-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 15:46:11,455-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154414.460, now=131106-154615.283, channel=TCPChannel [type: server, contact: 127.0.0.1:41092]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 15:46:15,283-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154414.460, now=131106-154615.283, channel=TCPChannel [type: server, contact: 127.0.0.1:41092]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 15:46:15,283-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154414.460, now=131106-154615.283, channel=TCPChannel [type: server, contact: 127.0.0.1:41092]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 15:46:15,283-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154414.460, now=131106-154615.283, channel=TCPChannel [type: server, contact: 127.0.0.1:41092]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 15:46:15,284-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154414.460, now=131106-154615.283, channel=TCPChannel [type: server, contact: 127.0.0.1:41092]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 15:46:15,284-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154414.460, now=131106-154615.283, channel=TCPChannel [type: server, contact: 127.0.0.1:41092]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 15:46:15,284-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154414.460, now=131106-154615.283, channel=TCPChannel [type: server, contact: 127.0.0.1:41092]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 15:46:15,284-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154414.460, now=131106-154615.283, channel=TCPChannel [type: server, contact: 127.0.0.1:41092]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 15:46:15,284-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154414.460, now=131106-154615.283, channel=TCPChannel [type: server, contact: 127.0.0.1:41092]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 15:46:15,284-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154414.460, now=131106-154615.283, channel=TCPChannel [type: server, contact: 127.0.0.1:41092]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 15:46:15,284-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41092] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154414.460, now=131106-154615.283, channel=TCPChannel [type: server, contact: 127.0.0.1:41092]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 15:46:15,284-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41092] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154414.460, now=131106-154615.283, channel=TCPChannel [type: server, contact: 127.0.0.1:41092]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 15:46:15,284-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154414.561, now=131106-154615.420, channel=TCPChannel [type: server, contact: 127.0.0.1:41093]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 15:46:15,420-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154414.561, now=131106-154615.420, channel=TCPChannel [type: server, contact: 127.0.0.1:41093]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 15:46:15,420-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154414.561, now=131106-154615.420, channel=TCPChannel [type: server, contact: 127.0.0.1:41093]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 15:46:15,420-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154414.561, now=131106-154615.420, channel=TCPChannel [type: server, contact: 127.0.0.1:41093]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 15:46:15,420-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154414.561, now=131106-154615.420, channel=TCPChannel [type: server, contact: 127.0.0.1:41093]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 15:46:15,420-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154414.561, now=131106-154615.420, channel=TCPChannel [type: server, contact: 127.0.0.1:41093]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 15:46:15,420-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154414.561, now=131106-154615.420, channel=TCPChannel [type: server, contact: 127.0.0.1:41093]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 15:46:15,420-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154414.561, now=131106-154615.420, channel=TCPChannel [type: server, contact: 127.0.0.1:41093]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 15:46:15,421-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154414.561, now=131106-154615.420, channel=TCPChannel [type: server, contact: 127.0.0.1:41093]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 15:46:15,421-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154414.561, now=131106-154615.420, channel=TCPChannel [type: server, contact: 127.0.0.1:41093]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 15:46:15,421-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41093] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154414.561, now=131106-154615.420, channel=TCPChannel [type: server, contact: 127.0.0.1:41093]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 15:46:15,421-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41093] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154414.561, now=131106-154615.420, channel=TCPChannel [type: server, contact: 127.0.0.1:41093]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 15:46:15,421-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154414.662, now=131106-154615.520, channel=TCPChannel [type: server, contact: 127.0.0.1:41094]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 15:46:15,520-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154414.662, now=131106-154615.520, channel=TCPChannel [type: server, contact: 127.0.0.1:41094]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 15:46:15,520-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154414.662, now=131106-154615.520, channel=TCPChannel [type: server, contact: 127.0.0.1:41094]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 15:46:15,520-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154414.662, now=131106-154615.520, channel=TCPChannel [type: server, contact: 127.0.0.1:41094]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 15:46:15,521-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41094] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154414.662, now=131106-154615.520, channel=TCPChannel [type: server, contact: 127.0.0.1:41094]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 15:46:15,521-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41094] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154414.662, now=131106-154615.520, channel=TCPChannel [type: server, contact: 127.0.0.1:41094]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 15:46:15,521-0600 INFO ChannelManager Channel already shut down
-No streams
-2013-11-06 15:46:17,654-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:46:27,662-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:46:37,669-0600 INFO Multiplexer No streams
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 15:46:40,086-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41097] for addition
-2013-11-06 15:46:40,086-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41097] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41097]
-2013-11-06 15:46:40,086-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41097]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fc9
-2013-11-06 15:46:40,117-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fc9
-MetaChannel [context: cps-41097, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41097]
-2013-11-06 15:46:40,117-0600 INFO MetaChannel MetaChannel [context: cps-41097, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41097]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 15:46:40,117-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 15:46:40,158-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 15:46:40,158-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 15:46:40,238-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 15:46:40,245-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 39.0) submitting urn:job-0-1383759274612-1383759274613: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 15:46:40,245-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 39.0) submitting urn:job-0-1383759274612-1383759274613: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 15:46:40,247-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 15:46:40,247-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-06 15:46:40,252-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7410, JOBSTATUS) unregistering (send)
-2013-11-06 15:46:40,252-0600 INFO RequestHandler Handler(tag: 7410, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 15:46:40,296-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41098] for addition
-2013-11-06 15:46:40,296-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41098] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41098]
-2013-11-06 15:46:40,296-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41098]
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7fc8
-2013-11-06 15:46:40,319-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7fc8
-MetaChannel [context: cps-41098, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41098]
-2013-11-06 15:46:40,319-0600 INFO MetaChannel MetaChannel [context: cps-41098, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41098]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-06 15:46:40,319-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-06 15:46:40,360-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 15:46:40,440-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 15:46:40,447-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-crush.mcs.anl.gov:0 (quality: 4.0) submitting urn:job-1-1383759274615-1383759274616: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 15:46:40,447-0600 INFO Cpu crush.mcs.anl.gov:0 (quality: 4.0) submitting urn:job-1-1383759274615-1383759274616: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 15:46:40,497-0600 DEBUG Cpu Active workerid=crush.mcs.anl.gov:000000
-Handler(tag: 39, JOBSTATUS) unregistering (send)
-2013-11-06 15:46:40,497-0600 INFO RequestHandler Handler(tag: 39, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 15:46:40,501-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41099] for addition
-2013-11-06 15:46:40,501-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41099] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41099]
-2013-11-06 15:46:40,501-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41099]
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7fc7
-2013-11-06 15:46:40,541-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7fc7
-MetaChannel [context: cps-41099, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41099]
-2013-11-06 15:46:40,541-0600 INFO MetaChannel MetaChannel [context: cps-41099, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41099]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-06 15:46:40,541-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-06 15:46:40,562-0600 INFO RequestHandler Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-06 15:46:40,642-0600 INFO RequestHandler Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-06 15:46:42,262-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 15:46:42,262-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:46:42,262-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:46:42,262-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Channel config: null -> {keepalive=null}
-2013-11-06 15:46:42,262-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-Handler(tag: 7411, JOBSTATUS) unregistering (send)
-2013-11-06 15:46:42,263-0600 INFO RequestHandler Handler(tag: 7411, JOBSTATUS) unregistering (send)
-TCPChannel [type: server, contact: 127.0.0.1:41097] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:46:42,263-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41097] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:46:42,263-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-null: Channel shut down
-2013-11-06 15:46:42,263-0600 INFO AbstractTCPChannel null: Channel shut down
-grind.mcs.anl.gov:0 (quality: 40.0) submitting urn:job-2-1383759274618-1383759274619: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 15:46:42,263-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 40.0) submitting urn:job-2-1383759274618-1383759274619: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-MetaChannel [context: cps-41097, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41097]] binding to NullChannel
-2013-11-06 15:46:42,263-0600 INFO MetaChannel MetaChannel [context: cps-41097, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41097]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41097]
-2013-11-06 15:46:42,264-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41097]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:46:42,264-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 15:46:42,264-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:46:42,264-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 15:46:42,264-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:46:42,264-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 15:46:42,265-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:46:42,265-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 15:46:42,265-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:46:42,265-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 15:46:42,265-0600 INFO ChannelManager Channel exception handled
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:46:42,265-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel config: null -> {keepalive=null}
-2013-11-06 15:46:42,266-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41099] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:46:42,266-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41099] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-06 15:46:42,266-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-41099, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41099]] binding to NullChannel
-2013-11-06 15:46:42,266-0600 INFO MetaChannel MetaChannel [context: cps-41099, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41099]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41099]
-2013-11-06 15:46:42,266-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41099]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:46:42,266-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 15:46:42,266-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:46:42,266-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 15:46:42,267-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 15:46:42,267-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 15:46:42,267-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 15:46:42,267-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41099] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 15:46:42,267-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41099] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 15:46:42,268-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41099]
-2013-11-06 15:46:42,268-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41099]
-Channel is closed. Removing.
-2013-11-06 15:46:42,268-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 15:46:42,268-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 15:46:42,268-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41097] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 15:46:42,268-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41097] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 15:46:42,268-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41097]
-2013-11-06 15:46:42,268-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41097]
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 15:46:42,269-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 15:46:42,270-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 15:46:42,271-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7412, JOBSTATUS) unregistering (send)
-2013-11-06 15:46:42,271-0600 INFO RequestHandler Handler(tag: 7412, JOBSTATUS) unregistering (send)
-2013-11-06 15:46:42,519-0600 DEBUG Cpu Completed
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:46:42,519-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-crush.mcs.anl.gov:0 jobTerminated
-2013-11-06 15:46:42,519-0600 INFO Cpu crush.mcs.anl.gov:0 jobTerminated
-Channel config: null -> {keepalive=null}
-2013-11-06 15:46:42,519-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-06 15:46:42,519-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-TCPChannel [type: server, contact: 127.0.0.1:41098] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:46:42,519-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41098] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:46:42,519-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-Handler(tag: 40, JOBSTATUS) unregistering (send)
-2013-11-06 15:46:42,519-0600 INFO RequestHandler Handler(tag: 40, JOBSTATUS) unregistering (send)
-2013-11-06 15:46:42,520-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-null: Channel shut down
-2013-11-06 15:46:42,520-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-41098, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41098]] binding to NullChannel
-2013-11-06 15:46:42,520-0600 INFO MetaChannel MetaChannel [context: cps-41098, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41098]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41098]
-2013-11-06 15:46:42,520-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41098]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:46:42,520-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 15:46:42,520-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:46:42,520-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 15:46:42,521-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:46:42,521-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 15:46:42,521-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:46:42,521-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 15:46:42,521-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:46:42,521-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 15:46:42,521-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 15:46:42,529-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 15:46:42,529-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 15:46:42,529-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41098] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 15:46:42,529-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41098] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 15:46:42,529-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41098]
-2013-11-06 15:46:42,529-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41098]
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 15:46:43,867-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 15:46:44,282-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 15:46:44,282-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 15:46:44,282-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 15:46:44,282-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-06 15:46:44,282-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7413, JOBSTATUS) unregistering (send)
-2013-11-06 15:46:44,282-0600 INFO RequestHandler Handler(tag: 7413, JOBSTATUS) unregistering (send)
-2013-11-06 15:46:44,282-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-No streams
-2013-11-06 15:46:47,684-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:46:47,929-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:46:57,692-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:47:07,700-0600 INFO Multiplexer No streams
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154513.491, now=131106-154714.348, channel=TCPChannel [type: server, contact: 127.0.0.1:41095]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 15:47:14,348-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154513.491, now=131106-154714.348, channel=TCPChannel [type: server, contact: 127.0.0.1:41095]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 15:47:14,348-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41095] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154513.491, now=131106-154714.348, channel=TCPChannel [type: server, contact: 127.0.0.1:41095]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 15:47:14,348-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41095] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154513.491, now=131106-154714.348, channel=TCPChannel [type: server, contact: 127.0.0.1:41095]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 15:47:14,349-0600 INFO ChannelManager Channel already shut down
-No streams
-2013-11-06 15:47:17,708-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:47:21,185-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 15:47:24,404-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 15:47:27,716-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:47:37,724-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:47:47,732-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:47:57,740-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 15:47:58,504-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154558.900, now=131106-154759.792, channel=TCPChannel [type: server, contact: 127.0.0.1:41096]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 15:47:59,792-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154558.900, now=131106-154759.792, channel=TCPChannel [type: server, contact: 127.0.0.1:41096]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 15:47:59,793-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41096] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154558.900, now=131106-154759.792, channel=TCPChannel [type: server, contact: 127.0.0.1:41096]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 15:47:59,793-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41096] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154558.900, now=131106-154759.792, channel=TCPChannel [type: server, contact: 127.0.0.1:41096]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 15:47:59,793-0600 INFO ChannelManager Channel already shut down
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:48:00,878-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:48:07,748-0600 INFO Multiplexer No streams
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 15:48:13,674-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41100] for addition
-2013-11-06 15:48:13,674-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41100] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41100]
-2013-11-06 15:48:13,674-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41100]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fc6
-2013-11-06 15:48:13,706-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fc6
-MetaChannel [context: cps-41100, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41100]
-2013-11-06 15:48:13,706-0600 INFO MetaChannel MetaChannel [context: cps-41100, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41100]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 15:48:13,706-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 15:48:13,767-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 15:48:13,767-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 15:48:13,847-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 15:48:13,883-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 41.0) submitting urn:job-0-1383759274621-1383759274622: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 15:48:13,884-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 41.0) submitting urn:job-0-1383759274621-1383759274622: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 15:48:13,884-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 15:48:13,884-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 15:48:13,908-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41101] for addition
-2013-11-06 15:48:13,908-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41101] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41101]
-2013-11-06 15:48:13,908-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41101]
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7fc5
-2013-11-06 15:48:13,948-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7fc5
-MetaChannel [context: cps-41101, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41101]
-2013-11-06 15:48:13,948-0600 INFO MetaChannel MetaChannel [context: cps-41101, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41101]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-06 15:48:13,948-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-06 15:48:13,976-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7414, JOBSTATUS) unregistering (send)
-2013-11-06 15:48:13,976-0600 INFO RequestHandler Handler(tag: 7414, JOBSTATUS) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-06 15:48:13,989-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 15:48:14,069-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 15:48:14,085-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-crush.mcs.anl.gov:0 (quality: 5.0) submitting urn:job-1-1383759274624-1383759274625: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 15:48:14,085-0600 INFO Cpu crush.mcs.anl.gov:0 (quality: 5.0) submitting urn:job-1-1383759274624-1383759274625: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 15:48:14,092-0600 DEBUG Cpu Active workerid=crush.mcs.anl.gov:000000
-Handler(tag: 41, JOBSTATUS) unregistering (send)
-2013-11-06 15:48:14,092-0600 INFO RequestHandler Handler(tag: 41, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 15:48:14,127-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41102] for addition
-2013-11-06 15:48:14,128-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41102] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41102]
-2013-11-06 15:48:14,128-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41102]
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7fc4
-2013-11-06 15:48:14,150-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7fc4
-MetaChannel [context: cps-41102, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41102]
-2013-11-06 15:48:14,150-0600 INFO MetaChannel MetaChannel [context: cps-41102, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41102]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-06 15:48:14,150-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-06 15:48:14,190-0600 INFO RequestHandler Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-06 15:48:14,271-0600 INFO RequestHandler Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-06 15:48:16,000-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 15:48:16,000-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 15:48:16,000-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:48:16,000-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 7415, JOBSTATUS) unregistering (send)
-2013-11-06 15:48:16,000-0600 INFO RequestHandler Handler(tag: 7415, JOBSTATUS) unregistering (send)
-2013-11-06 15:48:16,000-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-Channel config: null -> {keepalive=null}
-2013-11-06 15:48:16,001-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-grind.mcs.anl.gov:0 (quality: 42.0) submitting urn:job-2-1383759274627-1383759274628: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 15:48:16,001-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 42.0) submitting urn:job-2-1383759274627-1383759274628: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-TCPChannel [type: server, contact: 127.0.0.1:41100] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:48:16,001-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41100] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-06 15:48:16,002-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-41100, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41100]] binding to NullChannel
-2013-11-06 15:48:16,002-0600 INFO MetaChannel MetaChannel [context: cps-41100, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41100]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41100]
-2013-11-06 15:48:16,002-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41100]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:48:16,002-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 15:48:16,002-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:48:16,002-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 15:48:16,003-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:48:16,003-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 15:48:16,003-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:48:16,003-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 15:48:16,003-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:48:16,003-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 15:48:16,004-0600 INFO ChannelManager Channel exception handled
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:48:16,004-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel config: null -> {keepalive=null}
-2013-11-06 15:48:16,004-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41102] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:48:16,005-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41102] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-06 15:48:16,005-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-41102, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41102]] binding to NullChannel
-2013-11-06 15:48:16,005-0600 INFO MetaChannel MetaChannel [context: cps-41102, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41102]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41102]
-2013-11-06 15:48:16,005-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41102]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:48:16,005-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 15:48:16,005-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:48:16,005-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 15:48:16,006-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 15:48:16,016-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 15:48:16,016-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 15:48:16,016-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41102] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 15:48:16,016-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41102] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 15:48:16,017-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41102]
-2013-11-06 15:48:16,017-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41102]
-Channel is closed. Removing.
-2013-11-06 15:48:16,017-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 15:48:16,017-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 15:48:16,017-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41100] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 15:48:16,017-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41100] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 15:48:16,017-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41100]
-2013-11-06 15:48:16,017-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41100]
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 15:48:16,029-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 15:48:16,030-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 15:48:16,031-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7416, JOBSTATUS) unregistering (send)
-2013-11-06 15:48:16,031-0600 INFO RequestHandler Handler(tag: 7416, JOBSTATUS) unregistering (send)
-2013-11-06 15:48:16,112-0600 DEBUG Cpu Completed
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:48:16,112-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-crush.mcs.anl.gov:0 jobTerminated
-2013-11-06 15:48:16,112-0600 INFO Cpu crush.mcs.anl.gov:0 jobTerminated
-Channel config: null -> {keepalive=null}
-2013-11-06 15:48:16,112-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-06 15:48:16,112-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-TCPChannel [type: server, contact: 127.0.0.1:41101] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:48:16,112-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41101] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 42, JOBSTATUS) unregistering (send)
-2013-11-06 15:48:16,113-0600 INFO RequestHandler Handler(tag: 42, JOBSTATUS) unregistering (send)
-2013-11-06 15:48:16,112-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-null: Channel shut down
-2013-11-06 15:48:16,113-0600 INFO AbstractTCPChannel null: Channel shut down
-2013-11-06 15:48:16,113-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-MetaChannel [context: cps-41101, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41101]] binding to NullChannel
-2013-11-06 15:48:16,113-0600 INFO MetaChannel MetaChannel [context: cps-41101, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41101]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41101]
-2013-11-06 15:48:16,113-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41101]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:48:16,113-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 15:48:16,114-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:48:16,114-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 15:48:16,114-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:48:16,114-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 15:48:16,114-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:48:16,114-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 15:48:16,115-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:48:16,115-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 15:48:16,115-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 15:48:16,118-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 15:48:16,118-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 15:48:16,118-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41101] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 15:48:16,118-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41101] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 15:48:16,118-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41101]
-2013-11-06 15:48:16,118-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41101]
-No streams
-2013-11-06 15:48:17,763-0600 INFO Multiplexer No streams
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 15:48:18,049-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 15:48:18,049-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 15:48:18,049-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 15:48:18,049-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-06 15:48:18,049-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7417, JOBSTATUS) unregistering (send)
-2013-11-06 15:48:18,049-0600 INFO RequestHandler Handler(tag: 7417, JOBSTATUS) unregistering (send)
-2013-11-06 15:48:18,049-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-No streams
-2013-11-06 15:48:27,770-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:48:35,821-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:48:37,351-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:48:37,778-0600 INFO Multiplexer No streams
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154640.238, now=131106-154841.086, channel=TCPChannel [type: server, contact: 127.0.0.1:41097]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 15:48:41,086-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154640.238, now=131106-154841.086, channel=TCPChannel [type: server, contact: 127.0.0.1:41097]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 15:48:41,086-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154640.238, now=131106-154841.086, channel=TCPChannel [type: server, contact: 127.0.0.1:41097]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 15:48:41,086-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154640.238, now=131106-154841.086, channel=TCPChannel [type: server, contact: 127.0.0.1:41097]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 15:48:41,086-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154640.238, now=131106-154841.086, channel=TCPChannel [type: server, contact: 127.0.0.1:41097]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 15:48:41,086-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154640.238, now=131106-154841.086, channel=TCPChannel [type: server, contact: 127.0.0.1:41097]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 15:48:41,087-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154640.238, now=131106-154841.086, channel=TCPChannel [type: server, contact: 127.0.0.1:41097]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 15:48:41,087-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154640.238, now=131106-154841.086, channel=TCPChannel [type: server, contact: 127.0.0.1:41097]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 15:48:41,087-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154640.238, now=131106-154841.086, channel=TCPChannel [type: server, contact: 127.0.0.1:41097]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 15:48:41,087-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154640.238, now=131106-154841.086, channel=TCPChannel [type: server, contact: 127.0.0.1:41097]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 15:48:41,087-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41097] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154640.238, now=131106-154841.086, channel=TCPChannel [type: server, contact: 127.0.0.1:41097]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 15:48:41,087-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41097] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154640.238, now=131106-154841.086, channel=TCPChannel [type: server, contact: 127.0.0.1:41097]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 15:48:41,087-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154640.440, now=131106-154841.297, channel=TCPChannel [type: server, contact: 127.0.0.1:41098]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 15:48:41,297-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154640.440, now=131106-154841.297, channel=TCPChannel [type: server, contact: 127.0.0.1:41098]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 15:48:41,297-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154640.440, now=131106-154841.297, channel=TCPChannel [type: server, contact: 127.0.0.1:41098]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 15:48:41,297-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154640.440, now=131106-154841.297, channel=TCPChannel [type: server, contact: 127.0.0.1:41098]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 15:48:41,297-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154640.440, now=131106-154841.297, channel=TCPChannel [type: server, contact: 127.0.0.1:41098]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 15:48:41,297-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154640.440, now=131106-154841.297, channel=TCPChannel [type: server, contact: 127.0.0.1:41098]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 15:48:41,297-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154640.440, now=131106-154841.297, channel=TCPChannel [type: server, contact: 127.0.0.1:41098]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 15:48:41,297-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154640.440, now=131106-154841.297, channel=TCPChannel [type: server, contact: 127.0.0.1:41098]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 15:48:41,297-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154640.440, now=131106-154841.297, channel=TCPChannel [type: server, contact: 127.0.0.1:41098]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 15:48:41,297-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154640.440, now=131106-154841.297, channel=TCPChannel [type: server, contact: 127.0.0.1:41098]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 15:48:41,298-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41098] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154640.440, now=131106-154841.297, channel=TCPChannel [type: server, contact: 127.0.0.1:41098]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 15:48:41,298-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41098] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154640.440, now=131106-154841.297, channel=TCPChannel [type: server, contact: 127.0.0.1:41098]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 15:48:41,298-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154640.642, now=131106-154841.501, channel=TCPChannel [type: server, contact: 127.0.0.1:41099]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 15:48:41,501-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154640.642, now=131106-154841.501, channel=TCPChannel [type: server, contact: 127.0.0.1:41099]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 15:48:41,501-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154640.642, now=131106-154841.501, channel=TCPChannel [type: server, contact: 127.0.0.1:41099]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 15:48:41,501-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154640.642, now=131106-154841.501, channel=TCPChannel [type: server, contact: 127.0.0.1:41099]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 15:48:41,501-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41099] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154640.642, now=131106-154841.501, channel=TCPChannel [type: server, contact: 127.0.0.1:41099]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 15:48:41,502-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41099] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154640.642, now=131106-154841.501, channel=TCPChannel [type: server, contact: 127.0.0.1:41099]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 15:48:41,502-0600 INFO ChannelManager Channel already shut down
-No streams
-2013-11-06 15:48:47,786-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:48:57,794-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:49:07,802-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:49:13,139-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:49:13,825-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:49:17,809-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:49:27,817-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:49:37,825-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:49:47,832-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:49:50,299-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:49:50,457-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:49:57,840-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:50:07,848-0600 INFO Multiplexer No streams
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154813.847, now=131106-155014.674, channel=TCPChannel [type: server, contact: 127.0.0.1:41100]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 15:50:14,674-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154813.847, now=131106-155014.674, channel=TCPChannel [type: server, contact: 127.0.0.1:41100]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 15:50:14,674-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154813.847, now=131106-155014.674, channel=TCPChannel [type: server, contact: 127.0.0.1:41100]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 15:50:14,674-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154813.847, now=131106-155014.674, channel=TCPChannel [type: server, contact: 127.0.0.1:41100]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 15:50:14,674-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154813.847, now=131106-155014.674, channel=TCPChannel [type: server, contact: 127.0.0.1:41100]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 15:50:14,674-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154813.847, now=131106-155014.674, channel=TCPChannel [type: server, contact: 127.0.0.1:41100]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 15:50:14,675-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154813.847, now=131106-155014.674, channel=TCPChannel [type: server, contact: 127.0.0.1:41100]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 15:50:14,675-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154813.847, now=131106-155014.674, channel=TCPChannel [type: server, contact: 127.0.0.1:41100]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 15:50:14,675-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154813.847, now=131106-155014.674, channel=TCPChannel [type: server, contact: 127.0.0.1:41100]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 15:50:14,675-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154813.847, now=131106-155014.674, channel=TCPChannel [type: server, contact: 127.0.0.1:41100]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 15:50:14,675-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41100] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154813.847, now=131106-155014.674, channel=TCPChannel [type: server, contact: 127.0.0.1:41100]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 15:50:14,675-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41100] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154813.847, now=131106-155014.674, channel=TCPChannel [type: server, contact: 127.0.0.1:41100]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 15:50:14,675-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154814.069, now=131106-155014.909, channel=TCPChannel [type: server, contact: 127.0.0.1:41101]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 15:50:14,909-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154814.069, now=131106-155014.909, channel=TCPChannel [type: server, contact: 127.0.0.1:41101]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 15:50:14,909-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154814.069, now=131106-155014.909, channel=TCPChannel [type: server, contact: 127.0.0.1:41101]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 15:50:14,909-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154814.069, now=131106-155014.909, channel=TCPChannel [type: server, contact: 127.0.0.1:41101]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 15:50:14,909-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154814.069, now=131106-155014.909, channel=TCPChannel [type: server, contact: 127.0.0.1:41101]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 15:50:14,909-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154814.069, now=131106-155014.909, channel=TCPChannel [type: server, contact: 127.0.0.1:41101]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 15:50:14,909-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154814.069, now=131106-155014.909, channel=TCPChannel [type: server, contact: 127.0.0.1:41101]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 15:50:14,909-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154814.069, now=131106-155014.909, channel=TCPChannel [type: server, contact: 127.0.0.1:41101]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 15:50:14,910-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154814.069, now=131106-155014.909, channel=TCPChannel [type: server, contact: 127.0.0.1:41101]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 15:50:14,910-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154814.069, now=131106-155014.909, channel=TCPChannel [type: server, contact: 127.0.0.1:41101]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 15:50:14,910-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41101] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154814.069, now=131106-155014.909, channel=TCPChannel [type: server, contact: 127.0.0.1:41101]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 15:50:14,910-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41101] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154814.069, now=131106-155014.909, channel=TCPChannel [type: server, contact: 127.0.0.1:41101]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 15:50:14,910-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154814.271, now=131106-155015.127, channel=TCPChannel [type: server, contact: 127.0.0.1:41102]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 15:50:15,127-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154814.271, now=131106-155015.127, channel=TCPChannel [type: server, contact: 127.0.0.1:41102]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 15:50:15,127-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154814.271, now=131106-155015.127, channel=TCPChannel [type: server, contact: 127.0.0.1:41102]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 15:50:15,127-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154814.271, now=131106-155015.127, channel=TCPChannel [type: server, contact: 127.0.0.1:41102]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 15:50:15,128-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41102] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154814.271, now=131106-155015.127, channel=TCPChannel [type: server, contact: 127.0.0.1:41102]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 15:50:15,128-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41102] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-154814.271, now=131106-155015.127, channel=TCPChannel [type: server, contact: 127.0.0.1:41102]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 15:50:15,128-0600 INFO ChannelManager Channel already shut down
-No streams
-2013-11-06 15:50:17,856-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 15:50:26,774-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 15:50:27,775-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 15:50:27,863-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:50:37,870-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:50:47,878-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:50:57,886-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:51:03,247-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:51:05,094-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:51:07,894-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:51:17,901-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:51:27,909-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:51:37,917-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:51:39,721-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:51:42,412-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:51:47,924-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:51:57,932-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:52:07,939-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:52:16,195-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:52:17,947-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:52:19,730-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:52:27,955-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:52:37,962-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:52:47,970-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:52:52,669-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:52:57,048-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:52:57,977-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:53:07,985-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:53:17,993-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:53:28,000-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 15:53:29,144-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:53:34,366-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:53:38,008-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:53:48,016-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:53:58,024-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:54:05,617-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:54:08,032-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:54:11,684-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:54:18,040-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:54:28,048-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:54:38,055-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 15:54:42,092-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 15:54:48,063-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:54:49,002-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:54:58,071-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:55:08,078-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:55:18,086-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:55:18,565-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:55:26,320-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:55:28,094-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:55:38,102-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:55:48,109-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 15:55:55,040-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 15:55:58,117-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 15:56:03,638-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 15:56:08,125-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:56:18,132-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:56:28,140-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:56:31,513-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:56:38,148-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:56:40,956-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:56:48,156-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:56:58,163-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:57:07,987-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:57:08,171-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:57:18,179-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:57:18,274-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:57:28,187-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:57:38,195-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:57:44,461-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:57:48,203-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:57:55,592-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:57:58,211-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:58:08,219-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:58:18,226-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 15:58:20,936-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 15:58:28,234-0600 INFO Multiplexer No streams
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 15:58:32,427-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41108] for addition
-2013-11-06 15:58:32,427-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41108] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41108]
-2013-11-06 15:58:32,427-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41108]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fc3
-2013-11-06 15:58:32,467-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fc3
-MetaChannel [context: cps-41108, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41108]
-2013-11-06 15:58:32,468-0600 INFO MetaChannel MetaChannel [context: cps-41108, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41108]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 15:58:32,468-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 15:58:32,488-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 15:58:32,488-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 15:58:32,569-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 15:58:32,598-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 43.0) submitting urn:job-0-1383759274630-1383759274631: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 15:58:32,598-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 43.0) submitting urn:job-0-1383759274630-1383759274631: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 15:58:32,599-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 15:58:32,599-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-06 15:58:32,605-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7418, JOBSTATUS) unregistering (send)
-2013-11-06 15:58:32,606-0600 INFO RequestHandler Handler(tag: 7418, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 15:58:32,627-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41109] for addition
-2013-11-06 15:58:32,628-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41109] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41109]
-2013-11-06 15:58:32,628-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41109]
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7fc2
-2013-11-06 15:58:32,649-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7fc2
-MetaChannel [context: cps-41109, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41109]
-2013-11-06 15:58:32,650-0600 INFO MetaChannel MetaChannel [context: cps-41109, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41109]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-06 15:58:32,650-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-06 15:58:32,690-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 15:58:32,771-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 15:58:32,799-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-crush.mcs.anl.gov:0 (quality: 6.0) submitting urn:job-1-1383759274633-1383759274634: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 15:58:32,799-0600 INFO Cpu crush.mcs.anl.gov:0 (quality: 6.0) submitting urn:job-1-1383759274633-1383759274634: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 15:58:32,805-0600 DEBUG Cpu Active workerid=crush.mcs.anl.gov:000000
-Handler(tag: 43, JOBSTATUS) unregistering (send)
-2013-11-06 15:58:32,805-0600 INFO RequestHandler Handler(tag: 43, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 15:58:32,828-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41110] for addition
-2013-11-06 15:58:32,828-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41110] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41110]
-2013-11-06 15:58:32,828-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41110]
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7fc1
-2013-11-06 15:58:32,851-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7fc1
-MetaChannel [context: cps-41110, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41110]
-2013-11-06 15:58:32,851-0600 INFO MetaChannel MetaChannel [context: cps-41110, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41110]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-06 15:58:32,852-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-06 15:58:32,892-0600 INFO RequestHandler Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:58:32,910-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-06 15:58:32,973-0600 INFO RequestHandler Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-06 15:58:34,627-0600 DEBUG Cpu Completed
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:58:34,627-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 15:58:34,627-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-Channel config: null -> {keepalive=null}
-2013-11-06 15:58:34,627-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-06 15:58:34,628-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-TCPChannel [type: server, contact: 127.0.0.1:41108] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:58:34,628-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41108] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:58:34,628-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-null: Channel shut down
-2013-11-06 15:58:34,628-0600 INFO AbstractTCPChannel null: Channel shut down
-Handler(tag: 7419, JOBSTATUS) unregistering (send)
-2013-11-06 15:58:34,628-0600 INFO RequestHandler Handler(tag: 7419, JOBSTATUS) unregistering (send)
-MetaChannel [context: cps-41108, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41108]] binding to NullChannel
-2013-11-06 15:58:34,628-0600 INFO MetaChannel MetaChannel [context: cps-41108, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41108]] binding to NullChannel
-grind.mcs.anl.gov:0 (quality: 44.0) submitting urn:job-2-1383759274636-1383759274637: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 15:58:34,628-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 44.0) submitting urn:job-2-1383759274636-1383759274637: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41108]
-2013-11-06 15:58:34,629-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41108]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:58:34,629-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 15:58:34,629-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:58:34,629-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 15:58:34,629-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:58:34,630-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 15:58:34,630-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:58:34,630-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 15:58:34,630-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:58:34,630-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 15:58:34,630-0600 INFO ChannelManager Channel exception handled
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:58:34,630-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel config: null -> {keepalive=null}
-2013-11-06 15:58:34,631-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41110] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:58:34,631-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41110] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-06 15:58:34,631-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-41110, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41110]] binding to NullChannel
-2013-11-06 15:58:34,631-0600 INFO MetaChannel MetaChannel [context: cps-41110, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41110]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41110]
-2013-11-06 15:58:34,631-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41110]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:58:34,631-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 15:58:34,632-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:58:34,632-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 15:58:34,632-0600 INFO ChannelManager Channel exception handled
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 15:58:34,634-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 15:58:34,635-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 15:58:34,636-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7420, JOBSTATUS) unregistering (send)
-2013-11-06 15:58:34,636-0600 INFO RequestHandler Handler(tag: 7420, JOBSTATUS) unregistering (send)
-Channel is closed. Removing.
-2013-11-06 15:58:34,638-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 15:58:34,638-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 15:58:34,638-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41108] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 15:58:34,638-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41108] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 15:58:34,638-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41108]
-2013-11-06 15:58:34,638-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41108]
-Channel is closed. Removing.
-2013-11-06 15:58:34,638-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 15:58:34,638-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 15:58:34,638-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41110] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 15:58:34,639-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41110] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 15:58:34,639-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41110]
-2013-11-06 15:58:34,639-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41110]
-2013-11-06 15:58:34,940-0600 DEBUG Cpu Completed
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:58:34,940-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-crush.mcs.anl.gov:0 jobTerminated
-2013-11-06 15:58:34,940-0600 INFO Cpu crush.mcs.anl.gov:0 jobTerminated
-Channel config: null -> {keepalive=null}
-2013-11-06 15:58:34,941-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-06 15:58:34,941-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-TCPChannel [type: server, contact: 127.0.0.1:41109] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:58:34,941-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41109] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 44, JOBSTATUS) unregistering (send)
-2013-11-06 15:58:34,941-0600 INFO RequestHandler Handler(tag: 44, JOBSTATUS) unregistering (send)
-2013-11-06 15:58:34,941-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-null: Channel shut down
-2013-11-06 15:58:34,941-0600 INFO AbstractTCPChannel null: Channel shut down
-2013-11-06 15:58:34,941-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-MetaChannel [context: cps-41109, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41109]] binding to NullChannel
-2013-11-06 15:58:34,942-0600 INFO MetaChannel MetaChannel [context: cps-41109, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41109]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41109]
-2013-11-06 15:58:34,942-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41109]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:58:34,942-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 15:58:34,942-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:58:34,942-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 15:58:34,942-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:58:34,942-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 15:58:34,942-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:58:34,943-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 15:58:34,943-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 15:58:34,943-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 15:58:34,943-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 15:58:34,960-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 15:58:34,960-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 15:58:34,960-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41109] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 15:58:34,960-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41109] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 15:58:34,960-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41109]
-2013-11-06 15:58:34,960-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41109]
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 15:58:36,648-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 15:58:36,649-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 15:58:36,649-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 15:58:36,649-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-06 15:58:36,649-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7421, JOBSTATUS) unregistering (send)
-2013-11-06 15:58:36,649-0600 INFO RequestHandler Handler(tag: 7421, JOBSTATUS) unregistering (send)
-2013-11-06 15:58:36,649-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-No streams
-2013-11-06 15:58:38,249-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:58:48,258-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 15:58:57,409-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 15:58:58,266-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:59:08,274-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:59:10,228-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:59:18,282-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:59:28,290-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:59:33,883-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:59:38,298-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 15:59:47,546-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 15:59:48,306-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 15:59:58,314-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:00:08,322-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:00:10,357-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:00:18,330-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 16:00:24,865-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 16:00:28,338-0600 INFO Multiplexer No streams
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-155832.568, now=131106-160033.427, channel=TCPChannel [type: server, contact: 127.0.0.1:41108]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 16:00:33,427-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-155832.568, now=131106-160033.427, channel=TCPChannel [type: server, contact: 127.0.0.1:41108]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 16:00:33,427-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-155832.568, now=131106-160033.427, channel=TCPChannel [type: server, contact: 127.0.0.1:41108]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 16:00:33,428-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-155832.568, now=131106-160033.427, channel=TCPChannel [type: server, contact: 127.0.0.1:41108]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 16:00:33,428-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-155832.568, now=131106-160033.427, channel=TCPChannel [type: server, contact: 127.0.0.1:41108]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 16:00:33,428-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-155832.568, now=131106-160033.427, channel=TCPChannel [type: server, contact: 127.0.0.1:41108]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 16:00:33,428-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-155832.568, now=131106-160033.427, channel=TCPChannel [type: server, contact: 127.0.0.1:41108]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 16:00:33,428-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-155832.568, now=131106-160033.427, channel=TCPChannel [type: server, contact: 127.0.0.1:41108]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 16:00:33,428-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-155832.568, now=131106-160033.427, channel=TCPChannel [type: server, contact: 127.0.0.1:41108]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 16:00:33,428-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-155832.568, now=131106-160033.427, channel=TCPChannel [type: server, contact: 127.0.0.1:41108]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 16:00:33,428-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41108] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-155832.568, now=131106-160033.427, channel=TCPChannel [type: server, contact: 127.0.0.1:41108]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 16:00:33,428-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41108] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-155832.568, now=131106-160033.427, channel=TCPChannel [type: server, contact: 127.0.0.1:41108]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 16:00:33,429-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-155832.770, now=131106-160033.628, channel=TCPChannel [type: server, contact: 127.0.0.1:41109]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 16:00:33,628-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-155832.770, now=131106-160033.628, channel=TCPChannel [type: server, contact: 127.0.0.1:41109]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 16:00:33,628-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-155832.770, now=131106-160033.628, channel=TCPChannel [type: server, contact: 127.0.0.1:41109]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 16:00:33,628-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-155832.770, now=131106-160033.628, channel=TCPChannel [type: server, contact: 127.0.0.1:41109]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 16:00:33,628-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-155832.770, now=131106-160033.628, channel=TCPChannel [type: server, contact: 127.0.0.1:41109]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 16:00:33,628-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-155832.770, now=131106-160033.628, channel=TCPChannel [type: server, contact: 127.0.0.1:41109]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 16:00:33,628-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-155832.770, now=131106-160033.628, channel=TCPChannel [type: server, contact: 127.0.0.1:41109]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 16:00:33,629-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-155832.770, now=131106-160033.628, channel=TCPChannel [type: server, contact: 127.0.0.1:41109]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 16:00:33,629-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-155832.770, now=131106-160033.628, channel=TCPChannel [type: server, contact: 127.0.0.1:41109]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 16:00:33,629-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-155832.770, now=131106-160033.628, channel=TCPChannel [type: server, contact: 127.0.0.1:41109]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 16:00:33,629-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41109] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-155832.770, now=131106-160033.628, channel=TCPChannel [type: server, contact: 127.0.0.1:41109]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 16:00:33,629-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41109] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-155832.770, now=131106-160033.628, channel=TCPChannel [type: server, contact: 127.0.0.1:41109]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 16:00:33,629-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-155832.972, now=131106-160033.828, channel=TCPChannel [type: server, contact: 127.0.0.1:41110]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 16:00:33,828-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-155832.972, now=131106-160033.828, channel=TCPChannel [type: server, contact: 127.0.0.1:41110]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 16:00:33,829-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-155832.972, now=131106-160033.828, channel=TCPChannel [type: server, contact: 127.0.0.1:41110]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 16:00:33,829-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-155832.972, now=131106-160033.828, channel=TCPChannel [type: server, contact: 127.0.0.1:41110]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 16:00:33,829-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41110] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-155832.972, now=131106-160033.828, channel=TCPChannel [type: server, contact: 127.0.0.1:41110]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 16:00:33,829-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41110] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-155832.972, now=131106-160033.828, channel=TCPChannel [type: server, contact: 127.0.0.1:41110]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 16:00:33,829-0600 INFO ChannelManager Channel already shut down
-No streams
-2013-11-06 16:00:38,346-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:00:46,831-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:00:48,354-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:00:58,363-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:01:02,182-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:01:08,371-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:01:18,379-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 16:01:23,305-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 16:01:28,388-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:01:38,396-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 16:01:39,500-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 16:01:48,404-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:01:58,413-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:01:59,779-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:02:08,421-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:02:16,818-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:02:18,429-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:02:28,437-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 16:02:36,253-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 16:02:38,445-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:02:48,452-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 16:02:54,136-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 16:02:58,460-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:03:08,468-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:03:12,727-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:03:18,475-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:03:28,483-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:03:31,454-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:03:38,491-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:03:48,498-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:03:49,201-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:03:58,506-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:04:08,514-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 16:04:08,773-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 16:04:18,522-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:04:25,676-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:04:28,530-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:04:38,538-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:04:46,090-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:04:48,545-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:04:58,553-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:05:02,150-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:05:08,561-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:05:18,569-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:05:23,408-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:05:28,577-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:05:38,586-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:05:38,624-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:05:48,593-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:05:58,601-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:06:00,726-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:06:08,609-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:06:15,098-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:06:18,616-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:06:28,623-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:06:38,044-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:06:38,631-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:06:48,639-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:06:51,573-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:06:58,647-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:07:08,655-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 16:07:15,362-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 16:07:18,663-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:07:28,047-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:07:28,671-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:07:38,679-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:07:48,687-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:07:52,680-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:07:58,695-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 16:08:04,522-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 16:08:08,703-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:08:18,711-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:08:28,719-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 16:08:29,998-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 16:08:38,727-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:08:40,995-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:08:48,735-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:08:58,743-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:09:07,316-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:09:08,751-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:09:17,469-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:09:18,759-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:09:28,767-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:09:38,775-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:09:44,635-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:09:48,784-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:09:53,943-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:09:58,792-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:10:08,800-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:10:18,808-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:10:21,952-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:10:28,817-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 16:10:30,418-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 16:10:38,825-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:10:48,834-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:10:58,842-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:10:59,270-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:11:06,891-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:11:08,850-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:11:18,858-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:11:28,867-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:11:36,588-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:11:38,875-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 16:11:43,366-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 16:11:48,883-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:11:58,891-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:12:08,899-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:12:13,906-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:12:18,907-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:12:19,839-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:12:28,915-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:12:38,923-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:12:48,933-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:12:51,224-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:12:56,313-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:12:58,942-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:13:08,953-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:13:18,954-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:13:28,542-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:13:28,963-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:13:32,788-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:13:38,974-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:13:48,985-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:13:58,995-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:14:05,860-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:14:09,006-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:14:09,261-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:14:19,021-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:14:29,034-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:14:39,047-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 16:14:43,179-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 16:14:45,736-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 16:14:49,062-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:14:59,077-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:15:09,087-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:15:19,096-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 16:15:20,497-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:15:22,209-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:15:29,105-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:15:39,114-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:15:49,123-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:15:57,814-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 16:15:58,684-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 16:15:59,134-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:16:09,145-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:16:19,155-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:16:29,166-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 16:16:35,133-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:16:35,157-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:16:39,177-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:16:49,188-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:16:59,198-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:17:09,210-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:17:11,631-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 16:17:12,451-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 16:17:19,220-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:17:29,230-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:17:39,239-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 16:17:48,106-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 16:17:49,248-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:17:49,768-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:17:59,256-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:18:09,265-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:18:19,273-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:18:24,579-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:18:27,087-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:18:29,281-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:18:39,289-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:18:49,297-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:18:59,304-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:19:01,053-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:19:04,405-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:19:09,312-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:19:19,320-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:19:29,328-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 16:19:37,527-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 16:19:39,335-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:19:41,723-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:19:49,343-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:19:59,351-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:20:09,358-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:20:14,001-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 16:20:19,041-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 16:20:19,366-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:20:29,374-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:20:39,381-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:20:49,389-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 16:20:50,476-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:20:56,359-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:20:59,397-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:21:09,405-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:21:19,413-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 16:21:26,950-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 16:21:29,421-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:21:33,677-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:21:39,429-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:21:49,437-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:21:59,445-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:22:03,424-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:22:09,453-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 16:22:10,995-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 16:22:19,461-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:22:29,469-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:22:39,477-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:22:39,898-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:22:48,314-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:22:49,485-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:22:59,493-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:23:09,501-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:23:16,372-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:23:19,509-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:23:25,632-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:23:29,517-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:23:39,525-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:23:49,533-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:23:52,846-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:23:59,541-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:24:02,950-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:24:09,549-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:24:19,557-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:24:29,320-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:24:29,565-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:24:39,573-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:24:40,268-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:24:49,581-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:24:59,589-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 16:25:05,795-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 16:25:09,596-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:25:17,586-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:25:19,604-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:25:29,612-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:25:39,620-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 16:25:42,269-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 16:25:49,628-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:25:54,904-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:25:59,636-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:26:09,644-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:26:18,742-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:26:19,652-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:26:29,659-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:26:32,222-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:26:39,667-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:26:49,675-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 16:26:55,217-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 16:26:59,683-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:27:09,540-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:27:09,691-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:27:19,699-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:27:29,707-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:27:31,690-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:27:39,715-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:27:46,858-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:27:49,723-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:27:59,730-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 16:28:08,165-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 16:28:09,738-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:28:19,746-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:28:24,176-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:28:29,754-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:28:39,762-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:28:44,638-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:28:49,770-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:28:59,777-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:29:01,494-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:29:09,785-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:29:19,793-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 16:29:21,113-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 16:29:29,801-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:29:38,812-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:29:39,809-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:29:49,817-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:29:57,586-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:29:59,824-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:30:09,832-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:30:16,130-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:30:19,840-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:30:29,848-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:30:34,060-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:30:39,856-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:30:49,863-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:30:53,448-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:30:59,871-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:31:09,879-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 16:31:10,534-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 16:31:19,887-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:31:29,894-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:31:30,766-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:31:39,902-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:31:47,008-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:31:49,910-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:31:59,918-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:32:08,084-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:32:09,926-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:32:19,933-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:32:23,482-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:32:29,941-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:32:39,949-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:32:45,402-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:32:49,957-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 16:32:59,956-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 16:32:59,965-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:33:09,972-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:33:19,980-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:33:22,720-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:33:29,988-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:33:36,430-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:33:39,996-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:33:50,003-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:34:00,011-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:34:00,038-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:34:10,019-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 16:34:12,905-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 16:34:20,027-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:34:30,034-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 16:34:37,356-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 16:34:40,042-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:34:49,378-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:34:50,050-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:35:00,058-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:35:10,066-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-06 16:35:14,675-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-No streams
-2013-11-06 16:35:20,073-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:35:25,852-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:35:30,081-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:35:40,089-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:35:50,096-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:35:51,992-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:36:00,104-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:36:02,326-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:36:10,112-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:36:20,120-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:36:29,310-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:36:30,128-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:36:38,800-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:36:40,135-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:36:50,143-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:37:00,151-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:37:06,628-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:37:10,159-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:37:15,274-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:37:20,166-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:37:30,174-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:37:40,182-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:37:43,946-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:37:50,189-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:37:51,748-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:38:00,197-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:38:10,205-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:38:20,213-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:38:21,264-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:38:28,222-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:38:30,220-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:38:40,228-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:38:50,236-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:38:58,582-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:39:00,243-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:39:04,696-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:39:10,251-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:39:20,259-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:39:30,266-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-06 16:39:35,900-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-No streams
-2013-11-06 16:39:40,274-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 16:39:41,171-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 16:39:50,282-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:40:00,290-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:40:10,297-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:40:13,219-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:40:17,644-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:40:20,305-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:40:30,313-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:40:40,320-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:40:50,328-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:40:50,537-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:40:54,118-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:41:00,336-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:41:10,343-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:41:20,351-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:41:27,855-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:41:30,359-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:41:30,592-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:41:40,367-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:41:50,375-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:42:00,382-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-06 16:42:05,172-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:42:07,066-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:42:10,390-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:42:20,398-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:42:30,406-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:42:40,413-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:42:42,491-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 16:42:43,540-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 16:42:50,421-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:43:00,429-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:43:10,437-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:43:19,809-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:43:20,014-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:43:20,445-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:43:30,452-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:43:40,460-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:43:50,468-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 16:43:56,489-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-06 16:43:57,127-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-06 16:44:00,476-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:44:10,484-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:44:20,491-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:44:30,499-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:44:32,962-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:44:34,445-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:44:40,507-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:44:50,515-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:45:00,523-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:45:09,436-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:45:10,531-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:45:11,763-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:45:20,538-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:45:30,546-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:45:40,554-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 16:45:45,911-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:45:49,081-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:45:50,562-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:46:00,569-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:46:10,577-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:46:20,585-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:46:22,384-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-06 16:46:26,399-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-No streams
-2013-11-06 16:46:30,593-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:46:40,600-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:46:50,608-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:46:58,859-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:47:00,616-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:47:03,718-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:47:10,624-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:47:20,631-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:47:30,639-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 16:47:35,334-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 16:47:40,647-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:47:41,036-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:47:50,655-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:48:00,662-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:48:10,670-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:48:11,807-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-06 16:48:18,353-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-No streams
-2013-11-06 16:48:20,678-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:48:30,686-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:48:40,693-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:48:48,281-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:48:50,701-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:48:55,672-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:49:00,709-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:49:10,716-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:49:20,724-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:49:24,755-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:49:30,732-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:49:32,990-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:49:40,740-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:49:50,747-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:50:00,755-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 16:50:01,229-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-06 16:50:10,307-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-No streams
-2013-11-06 16:50:10,763-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:50:20,770-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:50:30,778-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:50:37,703-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:50:40,786-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:50:47,626-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:50:50,793-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:51:00,801-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:51:10,809-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:51:14,177-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:51:20,817-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:51:24,944-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:51:30,825-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:51:40,832-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 16:51:50,651-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 16:51:50,840-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:52:00,848-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:52:02,262-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:52:10,856-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:52:20,864-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:52:27,125-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:52:30,871-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-06 16:52:39,580-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-No streams
-2013-11-06 16:52:40,879-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:52:50,887-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:53:00,895-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:53:03,600-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:53:10,903-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:53:16,899-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:53:20,910-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:53:30,918-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:53:40,074-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:53:40,926-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:53:50,934-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:53:54,217-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:54:00,941-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:54:10,949-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:54:16,548-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:54:20,957-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:54:30,965-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-06 16:54:31,534-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-No streams
-2013-11-06 16:54:40,973-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:54:50,980-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 16:54:53,022-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 16:55:00,988-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:55:08,853-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:55:10,996-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:55:21,003-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:55:29,496-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:55:31,011-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:55:41,019-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:55:46,171-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:55:51,026-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:56:01,034-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:56:05,970-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:56:11,042-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:56:21,050-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:56:23,489-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:56:31,057-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:56:41,065-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:56:42,444-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:56:51,073-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:57:00,807-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:57:01,081-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:57:11,089-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:57:18,918-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:57:21,096-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:57:31,104-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:57:38,125-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:57:41,112-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:57:51,120-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 16:57:55,393-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 16:58:01,128-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:58:11,136-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:58:15,443-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:58:21,144-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:58:31,151-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:58:31,866-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:58:41,159-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:58:51,167-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:58:52,761-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:59:01,175-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 16:59:08,341-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 16:59:11,183-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:59:21,191-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 16:59:30,079-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 16:59:31,199-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 16:59:41,206-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 16:59:44,815-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 16:59:51,214-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:00:01,222-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:00:07,397-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:00:11,230-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:00:21,237-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:00:21,288-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:00:31,245-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:00:41,253-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-06 17:00:44,716-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-06 17:00:51,261-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 17:00:57,762-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 17:01:01,268-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:01:11,276-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:01:21,284-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:01:22,034-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:01:31,292-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:01:34,236-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:01:41,300-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:01:51,307-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:01:59,352-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:02:01,315-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 17:02:10,711-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 17:02:11,323-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:02:21,331-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:02:31,338-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-06 17:02:36,670-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-06 17:02:41,346-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:02:47,184-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:02:51,354-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:03:01,362-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:03:11,369-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:03:13,988-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:03:21,377-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:03:23,658-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:03:31,385-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:03:41,393-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:03:51,306-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:03:51,400-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:04:00,132-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:04:01,408-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:04:11,416-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:04:21,424-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-06 17:04:28,624-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-06 17:04:31,431-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:04:36,606-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:04:41,439-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:04:51,447-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:05:01,454-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-06 17:05:05,943-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-No streams
-2013-11-06 17:05:11,462-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 17:05:13,081-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 17:05:21,470-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:05:31,478-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:05:41,485-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:05:43,260-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:05:49,554-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:05:51,493-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:06:01,501-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:06:11,508-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:06:20,578-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:06:21,516-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 17:06:26,029-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 17:06:31,524-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:06:41,532-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:06:51,542-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-06 17:06:57,897-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-No streams
-2013-11-06 17:07:01,551-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 17:07:02,503-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 17:07:11,561-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:07:21,571-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:07:31,583-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:07:35,215-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 17:07:38,977-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 17:07:41,592-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:07:51,602-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:08:01,614-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:08:11,625-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:08:12,533-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 17:08:15,451-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 17:08:21,637-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:08:31,652-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:08:41,669-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-06 17:08:49,852-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-No streams
-2013-11-06 17:08:51,681-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:08:51,924-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:09:01,693-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:09:11,707-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:09:21,722-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-06 17:09:27,170-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 17:09:28,399-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 17:09:31,733-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:09:41,744-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:09:51,754-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:10:01,764-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:10:04,487-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:10:04,872-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:10:11,774-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:10:21,783-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:10:31,794-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:10:41,346-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:10:41,803-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:10:41,806-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:10:51,813-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:11:01,822-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:11:11,832-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:11:17,820-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-06 17:11:19,124-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-No streams
-2013-11-06 17:11:21,841-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:11:31,852-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:11:41,862-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:11:51,874-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 17:11:54,295-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:11:56,441-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:12:01,883-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:12:11,892-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:12:21,900-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:12:30,768-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:12:31,908-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:12:33,759-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:12:41,916-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:12:51,924-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:13:01,932-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 17:13:07,243-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:13:11,077-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:13:11,940-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:13:21,949-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:13:31,957-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:13:41,965-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:13:43,716-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-06 17:13:48,395-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-06 17:13:51,973-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:14:01,982-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:14:11,990-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:14:20,190-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:14:21,998-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:14:25,713-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:14:32,006-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:14:42,015-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:14:52,023-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 17:14:56,665-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 17:15:02,031-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:15:03,031-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:15:12,040-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:15:22,048-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:15:32,056-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:15:33,138-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:15:40,349-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:15:42,064-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:15:52,073-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:16:02,081-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:16:09,612-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:16:12,089-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:16:17,667-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:16:22,097-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:16:32,106-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:16:42,114-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:16:46,086-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:16:52,122-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:16:54,985-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:17:02,130-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:17:12,138-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:17:22,147-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 17:17:22,560-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 17:17:32,155-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-06 17:17:32,302-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-No streams
-2013-11-06 17:17:42,163-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:17:52,171-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:17:59,034-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:18:02,179-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:18:09,621-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:18:12,187-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:18:22,195-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:18:32,204-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 17:18:35,509-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 17:18:42,212-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-06 17:18:46,939-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-06 17:18:52,220-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:19:02,228-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:19:11,982-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:19:12,236-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:19:22,244-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:19:24,257-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:19:32,253-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:19:42,261-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:19:48,456-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:19:52,269-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-06 17:20:01,575-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-06 17:20:02,277-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:20:12,285-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:20:22,293-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:20:24,930-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:20:32,301-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-06 17:20:38,894-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-No streams
-2013-11-06 17:20:42,309-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:20:52,317-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-06 17:21:01,405-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-No streams
-2013-11-06 17:21:02,326-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:21:12,334-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:21:16,211-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:21:22,342-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:21:32,350-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:21:37,879-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:21:42,359-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:21:52,367-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:21:53,529-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:22:02,375-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:22:12,383-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:22:14,353-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:22:22,391-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:22:30,847-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:22:32,400-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:22:42,408-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 17:22:50,828-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 17:22:52,416-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:23:02,424-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-06 17:23:08,165-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-06 17:23:12,433-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:23:22,441-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:23:27,301-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:23:32,449-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:23:42,457-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:23:45,483-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:23:52,465-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:24:02,473-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:24:03,776-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:24:12,482-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:24:22,490-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:24:22,801-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:24:32,498-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:24:40,250-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:24:42,506-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:24:52,514-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-06 17:25:00,120-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-No streams
-2013-11-06 17:25:02,522-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:25:12,531-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:25:16,724-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:25:22,539-0600 INFO Multiplexer No streams
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:25:27,672-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41145] for addition
-2013-11-06 17:25:27,672-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41145] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41145]
-2013-11-06 17:25:27,672-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41145]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fc0
-2013-11-06 17:25:27,695-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fc0
-MetaChannel [context: cps-41145, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41145]
-2013-11-06 17:25:27,695-0600 INFO MetaChannel MetaChannel [context: cps-41145, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41145]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:25:27,695-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 17:25:27,736-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:25:27,736-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 17:25:27,817-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 17:25:27,845-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 45.0) submitting urn:job-0-1383759274639-1383759274640: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:25:27,845-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 45.0) submitting urn:job-0-1383759274639-1383759274640: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:25:27,845-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 17:25:27,845-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:25:27,876-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41146] for addition
-2013-11-06 17:25:27,876-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41146] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41146]
-2013-11-06 17:25:27,876-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41146]
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7fbf
-2013-11-06 17:25:27,897-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7fbf
-MetaChannel [context: cps-41146, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41146]
-2013-11-06 17:25:27,897-0600 INFO MetaChannel MetaChannel [context: cps-41146, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41146]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:25:27,898-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:25:27,938-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:25:27,963-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7422, JOBSTATUS) unregistering (send)
-2013-11-06 17:25:27,964-0600 INFO RequestHandler Handler(tag: 7422, JOBSTATUS) unregistering (send)
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 17:25:28,019-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 17:25:28,046-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-crush.mcs.anl.gov:0 (quality: 7.0) submitting urn:job-1-1383759274642-1383759274643: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:25:28,046-0600 INFO Cpu crush.mcs.anl.gov:0 (quality: 7.0) submitting urn:job-1-1383759274642-1383759274643: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:25:28,052-0600 DEBUG Cpu Active workerid=crush.mcs.anl.gov:000000
-Handler(tag: 45, JOBSTATUS) unregistering (send)
-2013-11-06 17:25:28,052-0600 INFO RequestHandler Handler(tag: 45, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:25:28,076-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41147] for addition
-2013-11-06 17:25:28,076-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41147] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41147]
-2013-11-06 17:25:28,076-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41147]
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7fbe
-2013-11-06 17:25:28,103-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7fbe
-MetaChannel [context: cps-41147, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41147]
-2013-11-06 17:25:28,103-0600 INFO MetaChannel MetaChannel [context: cps-41147, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41147]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:25:28,103-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:25:28,144-0600 INFO RequestHandler Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-06 17:25:28,224-0600 INFO RequestHandler Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-06 17:25:30,004-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:25:30,004-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:25:30,004-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:25:30,004-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Channel config: null -> {keepalive=null}
-2013-11-06 17:25:30,004-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-Handler(tag: 7423, JOBSTATUS) unregistering (send)
-2013-11-06 17:25:30,004-0600 INFO RequestHandler Handler(tag: 7423, JOBSTATUS) unregistering (send)
-TCPChannel [type: server, contact: 127.0.0.1:41145] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:25:30,004-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41145] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:25:30,004-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-null: Channel shut down
-2013-11-06 17:25:30,005-0600 INFO AbstractTCPChannel null: Channel shut down
-grind.mcs.anl.gov:0 (quality: 46.0) submitting urn:job-2-1383759274645-1383759274646: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:25:30,005-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 46.0) submitting urn:job-2-1383759274645-1383759274646: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-MetaChannel [context: cps-41145, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41145]] binding to NullChannel
-2013-11-06 17:25:30,005-0600 INFO MetaChannel MetaChannel [context: cps-41145, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41145]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41145]
-2013-11-06 17:25:30,005-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41145]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:25:30,005-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:25:30,006-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:25:30,006-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:25:30,006-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:25:30,006-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:25:30,006-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:25:30,006-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:25:30,007-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:25:30,007-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:25:30,007-0600 INFO ChannelManager Channel exception handled
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:25:30,007-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:25:30,007-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41147] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:25:30,007-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41147] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-06 17:25:30,008-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-41147, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41147]] binding to NullChannel
-2013-11-06 17:25:30,008-0600 INFO MetaChannel MetaChannel [context: cps-41147, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41147]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41147]
-2013-11-06 17:25:30,008-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41147]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:25:30,008-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:25:30,008-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:25:30,008-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:25:30,008-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 17:25:30,009-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:25:30,009-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:25:30,010-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41147] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:25:30,010-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41147] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:25:30,010-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41147]
-2013-11-06 17:25:30,010-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41147]
-Channel is closed. Removing.
-2013-11-06 17:25:30,010-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:25:30,010-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:25:30,010-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41145] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:25:30,010-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41145] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:25:30,010-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41145]
-2013-11-06 17:25:30,011-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41145]
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:25:30,020-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:25:30,020-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:25:30,021-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7424, JOBSTATUS) unregistering (send)
-2013-11-06 17:25:30,021-0600 INFO RequestHandler Handler(tag: 7424, JOBSTATUS) unregistering (send)
-2013-11-06 17:25:30,345-0600 DEBUG Cpu Completed
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:25:30,345-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-crush.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:25:30,345-0600 INFO Cpu crush.mcs.anl.gov:0 jobTerminated
-Channel config: null -> {keepalive=null}
-2013-11-06 17:25:30,345-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-06 17:25:30,345-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-TCPChannel [type: server, contact: 127.0.0.1:41146] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:25:30,345-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41146] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 46, JOBSTATUS) unregistering (send)
-2013-11-06 17:25:30,345-0600 INFO RequestHandler Handler(tag: 46, JOBSTATUS) unregistering (send)
-2013-11-06 17:25:30,345-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-null: Channel shut down
-2013-11-06 17:25:30,346-0600 INFO AbstractTCPChannel null: Channel shut down
-2013-11-06 17:25:30,346-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-MetaChannel [context: cps-41146, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41146]] binding to NullChannel
-2013-11-06 17:25:30,346-0600 INFO MetaChannel MetaChannel [context: cps-41146, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41146]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41146]
-2013-11-06 17:25:30,346-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41146]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:25:30,346-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:25:30,347-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:25:30,347-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:25:30,347-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:25:30,347-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:25:30,347-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:25:30,347-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:25:30,347-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:25:30,347-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:25:30,348-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 17:25:30,352-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:25:30,352-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:25:30,352-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41146] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:25:30,352-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41146] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:25:30,352-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41146]
-2013-11-06 17:25:30,352-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41146]
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:25:32,039-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:25:32,039-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:25:32,039-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:25:32,039-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-06 17:25:32,039-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7425, JOBSTATUS) unregistering (send)
-2013-11-06 17:25:32,039-0600 INFO RequestHandler Handler(tag: 7425, JOBSTATUS) unregistering (send)
-2013-11-06 17:25:32,039-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-No streams
-2013-11-06 17:25:32,558-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:25:37,437-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:25:42,567-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:25:52,575-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:25:53,198-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:26:02,583-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:26:12,591-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:26:14,755-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:26:22,599-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:26:29,672-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:26:32,608-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:26:42,616-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:26:52,073-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:26:52,624-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:27:02,632-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:27:06,146-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:27:12,640-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:27:22,648-0600 INFO Multiplexer No streams
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172527.816, now=131106-172728.672, channel=TCPChannel [type: server, contact: 127.0.0.1:41145]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:27:28,672-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172527.816, now=131106-172728.672, channel=TCPChannel [type: server, contact: 127.0.0.1:41145]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:27:28,672-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172527.816, now=131106-172728.672, channel=TCPChannel [type: server, contact: 127.0.0.1:41145]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:27:28,673-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172527.816, now=131106-172728.672, channel=TCPChannel [type: server, contact: 127.0.0.1:41145]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:27:28,673-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172527.816, now=131106-172728.672, channel=TCPChannel [type: server, contact: 127.0.0.1:41145]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:27:28,673-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172527.816, now=131106-172728.672, channel=TCPChannel [type: server, contact: 127.0.0.1:41145]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:27:28,673-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172527.816, now=131106-172728.672, channel=TCPChannel [type: server, contact: 127.0.0.1:41145]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:27:28,673-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172527.816, now=131106-172728.672, channel=TCPChannel [type: server, contact: 127.0.0.1:41145]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:27:28,673-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172527.816, now=131106-172728.672, channel=TCPChannel [type: server, contact: 127.0.0.1:41145]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:27:28,673-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172527.816, now=131106-172728.672, channel=TCPChannel [type: server, contact: 127.0.0.1:41145]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:27:28,673-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41145] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172527.816, now=131106-172728.672, channel=TCPChannel [type: server, contact: 127.0.0.1:41145]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:27:28,673-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41145] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172527.816, now=131106-172728.672, channel=TCPChannel [type: server, contact: 127.0.0.1:41145]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:27:28,674-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172528.018, now=131106-172728.876, channel=TCPChannel [type: server, contact: 127.0.0.1:41146]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:27:28,876-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172528.018, now=131106-172728.876, channel=TCPChannel [type: server, contact: 127.0.0.1:41146]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:27:28,876-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172528.018, now=131106-172728.876, channel=TCPChannel [type: server, contact: 127.0.0.1:41146]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:27:28,876-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172528.018, now=131106-172728.876, channel=TCPChannel [type: server, contact: 127.0.0.1:41146]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:27:28,876-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172528.018, now=131106-172728.876, channel=TCPChannel [type: server, contact: 127.0.0.1:41146]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:27:28,876-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172528.018, now=131106-172728.876, channel=TCPChannel [type: server, contact: 127.0.0.1:41146]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:27:28,877-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172528.018, now=131106-172728.876, channel=TCPChannel [type: server, contact: 127.0.0.1:41146]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:27:28,877-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172528.018, now=131106-172728.876, channel=TCPChannel [type: server, contact: 127.0.0.1:41146]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:27:28,877-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172528.018, now=131106-172728.876, channel=TCPChannel [type: server, contact: 127.0.0.1:41146]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:27:28,877-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172528.018, now=131106-172728.876, channel=TCPChannel [type: server, contact: 127.0.0.1:41146]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:27:28,877-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41146] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172528.018, now=131106-172728.876, channel=TCPChannel [type: server, contact: 127.0.0.1:41146]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:27:28,877-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41146] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172528.018, now=131106-172728.876, channel=TCPChannel [type: server, contact: 127.0.0.1:41146]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:27:28,877-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172528.224, now=131106-172729.076, channel=TCPChannel [type: server, contact: 127.0.0.1:41147]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:27:29,076-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172528.224, now=131106-172729.076, channel=TCPChannel [type: server, contact: 127.0.0.1:41147]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:27:29,077-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172528.224, now=131106-172729.076, channel=TCPChannel [type: server, contact: 127.0.0.1:41147]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:27:29,077-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172528.224, now=131106-172729.076, channel=TCPChannel [type: server, contact: 127.0.0.1:41147]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:27:29,077-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41147] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172528.224, now=131106-172729.076, channel=TCPChannel [type: server, contact: 127.0.0.1:41147]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:27:29,077-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41147] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172528.224, now=131106-172729.076, channel=TCPChannel [type: server, contact: 127.0.0.1:41147]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:27:29,077-0600 INFO ChannelManager Channel already shut down
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:27:29,391-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:27:32,656-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:27:42,620-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:27:42,664-0600 INFO Multiplexer No streams
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:27:52,126-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41148] for addition
-2013-11-06 17:27:52,126-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41148] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41148]
-2013-11-06 17:27:52,126-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41148]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fbd
-2013-11-06 17:27:52,152-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fbd
-MetaChannel [context: cps-41148, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41148]
-2013-11-06 17:27:52,152-0600 INFO MetaChannel MetaChannel [context: cps-41148, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41148]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:27:52,152-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 17:27:52,193-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:27:52,193-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 17:27:52,274-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 17:27:52,296-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 47.0) submitting urn:job-0-1383759274648-1383759274649: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:27:52,297-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 47.0) submitting urn:job-0-1383759274648-1383759274649: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:27:52,297-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 17:27:52,297-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-06 17:27:52,303-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7426, JOBSTATUS) unregistering (send)
-2013-11-06 17:27:52,303-0600 INFO RequestHandler Handler(tag: 7426, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:27:52,332-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41149] for addition
-2013-11-06 17:27:52,332-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41149] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41149]
-2013-11-06 17:27:52,332-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41149]
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7fbc
-2013-11-06 17:27:52,354-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7fbc
-MetaChannel [context: cps-41149, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41149]
-2013-11-06 17:27:52,354-0600 INFO MetaChannel MetaChannel [context: cps-41149, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41149]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:27:52,354-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:27:52,395-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 17:27:52,476-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 17:27:52,498-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-crush.mcs.anl.gov:0 (quality: 8.0) submitting urn:job-1-1383759274651-1383759274652: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:27:52,498-0600 INFO Cpu crush.mcs.anl.gov:0 (quality: 8.0) submitting urn:job-1-1383759274651-1383759274652: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:27:52,504-0600 DEBUG Cpu Active workerid=crush.mcs.anl.gov:000000
-Handler(tag: 47, JOBSTATUS) unregistering (send)
-2013-11-06 17:27:52,505-0600 INFO RequestHandler Handler(tag: 47, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:27:52,537-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41150] for addition
-2013-11-06 17:27:52,537-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41150] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41150]
-2013-11-06 17:27:52,537-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41150]
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7fbb
-2013-11-06 17:27:52,576-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7fbb
-MetaChannel [context: cps-41150, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41150]
-2013-11-06 17:27:52,576-0600 INFO MetaChannel MetaChannel [context: cps-41150, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41150]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:27:52,577-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:27:52,597-0600 INFO RequestHandler Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-Avg stream buf: 81
-2013-11-06 17:27:52,678-0600 INFO Multiplexer Avg stream buf: 81
-Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-06 17:27:52,678-0600 INFO RequestHandler Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-06 17:27:54,325-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:27:54,325-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:27:54,325-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:27:54,325-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 7427, JOBSTATUS) unregistering (send)
-2013-11-06 17:27:54,325-0600 INFO RequestHandler Handler(tag: 7427, JOBSTATUS) unregistering (send)
-2013-11-06 17:27:54,325-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-Channel config: null -> {keepalive=null}
-2013-11-06 17:27:54,326-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-grind.mcs.anl.gov:0 (quality: 48.0) submitting urn:job-2-1383759274654-1383759274655: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:27:54,326-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 48.0) submitting urn:job-2-1383759274654-1383759274655: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-TCPChannel [type: server, contact: 127.0.0.1:41148] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:27:54,326-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41148] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-06 17:27:54,327-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-41148, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41148]] binding to NullChannel
-2013-11-06 17:27:54,327-0600 INFO MetaChannel MetaChannel [context: cps-41148, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41148]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41148]
-2013-11-06 17:27:54,327-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41148]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:27:54,327-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:27:54,327-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:27:54,327-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:27:54,328-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:27:54,328-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:27:54,328-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:27:54,328-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:27:54,328-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:27:54,328-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:27:54,328-0600 INFO ChannelManager Channel exception handled
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:27:54,329-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:27:54,329-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41150] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:27:54,329-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41150] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-06 17:27:54,329-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-41150, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41150]] binding to NullChannel
-2013-11-06 17:27:54,329-0600 INFO MetaChannel MetaChannel [context: cps-41150, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41150]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41150]
-2013-11-06 17:27:54,329-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41150]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:27:54,329-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:27:54,330-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:27:54,330-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:27:54,330-0600 INFO ChannelManager Channel exception handled
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:27:54,332-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:27:54,333-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:27:54,333-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7428, JOBSTATUS) unregistering (send)
-2013-11-06 17:27:54,333-0600 INFO RequestHandler Handler(tag: 7428, JOBSTATUS) unregistering (send)
-Channel is closed. Removing.
-2013-11-06 17:27:54,343-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:27:54,343-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:27:54,343-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41150] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:27:54,344-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41150] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:27:54,344-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41150]
-2013-11-06 17:27:54,344-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41150]
-Channel is closed. Removing.
-2013-11-06 17:27:54,344-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:27:54,344-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:27:54,344-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41148] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:27:54,344-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41148] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:27:54,344-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41148]
-2013-11-06 17:27:54,344-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41148]
-2013-11-06 17:27:54,524-0600 DEBUG Cpu Completed
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:27:54,524-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-crush.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:27:54,524-0600 INFO Cpu crush.mcs.anl.gov:0 jobTerminated
-Channel config: null -> {keepalive=null}
-2013-11-06 17:27:54,524-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-06 17:27:54,525-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-TCPChannel [type: server, contact: 127.0.0.1:41149] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:27:54,525-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41149] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 48, JOBSTATUS) unregistering (send)
-2013-11-06 17:27:54,525-0600 INFO RequestHandler Handler(tag: 48, JOBSTATUS) unregistering (send)
-2013-11-06 17:27:54,525-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-null: Channel shut down
-2013-11-06 17:27:54,525-0600 INFO AbstractTCPChannel null: Channel shut down
-2013-11-06 17:27:54,525-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-MetaChannel [context: cps-41149, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41149]] binding to NullChannel
-2013-11-06 17:27:54,525-0600 INFO MetaChannel MetaChannel [context: cps-41149, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41149]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41149]
-2013-11-06 17:27:54,526-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41149]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:27:54,526-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:27:54,526-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:27:54,526-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:27:54,526-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:27:54,526-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:27:54,526-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:27:54,526-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:27:54,527-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:27:54,527-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:27:54,527-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 17:27:54,545-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:27:54,545-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:27:54,545-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41149] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:27:54,545-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41149] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:27:54,546-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41149]
-2013-11-06 17:27:54,546-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41149]
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:27:56,345-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:27:56,345-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:27:56,345-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:27:56,345-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-06 17:27:56,345-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7429, JOBSTATUS) unregistering (send)
-2013-11-06 17:27:56,345-0600 INFO RequestHandler Handler(tag: 7429, JOBSTATUS) unregistering (send)
-2013-11-06 17:27:56,345-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-No streams
-2013-11-06 17:28:02,688-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-06 17:28:06,708-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-No streams
-2013-11-06 17:28:12,695-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 17:28:19,095-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 17:28:22,702-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:28:32,710-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:28:42,718-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:28:44,027-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:28:52,726-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:28:55,568-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:29:02,734-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:29:12,742-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-06 17:29:21,345-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:29:22,081-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41151] for addition
-2013-11-06 17:29:22,081-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41151] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41151]
-2013-11-06 17:29:22,081-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41151]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fba
-2013-11-06 17:29:22,109-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fba
-MetaChannel [context: cps-41151, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41151]
-2013-11-06 17:29:22,109-0600 INFO MetaChannel MetaChannel [context: cps-41151, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41151]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:29:22,109-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:29:22,150-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 17:29:22,150-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 17:29:22,230-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 17:29:22,243-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 49.0) submitting urn:job-0-1383759274657-1383759274658: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:29:22,243-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 49.0) submitting urn:job-0-1383759274657-1383759274658: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:29:22,243-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 17:29:22,243-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-06 17:29:22,249-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7430, JOBSTATUS) unregistering (send)
-2013-11-06 17:29:22,250-0600 INFO RequestHandler Handler(tag: 7430, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:29:22,288-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41152] for addition
-2013-11-06 17:29:22,288-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41152] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41152]
-2013-11-06 17:29:22,288-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41152]
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7fb9
-2013-11-06 17:29:22,311-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7fb9
-MetaChannel [context: cps-41152, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41152]
-2013-11-06 17:29:22,311-0600 INFO MetaChannel MetaChannel [context: cps-41152, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41152]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:29:22,311-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:29:22,372-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 17:29:22,452-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 17:29:22,494-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-crush.mcs.anl.gov:0 (quality: 9.0) submitting urn:job-1-1383759274660-1383759274661: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:29:22,494-0600 INFO Cpu crush.mcs.anl.gov:0 (quality: 9.0) submitting urn:job-1-1383759274660-1383759274661: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:29:22,501-0600 DEBUG Cpu Active workerid=crush.mcs.anl.gov:000000
-Handler(tag: 49, JOBSTATUS) unregistering (send)
-2013-11-06 17:29:22,501-0600 INFO RequestHandler Handler(tag: 49, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:29:22,513-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41153] for addition
-2013-11-06 17:29:22,513-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41153] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41153]
-2013-11-06 17:29:22,513-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41153]
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7fb8
-2013-11-06 17:29:22,553-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7fb8
-MetaChannel [context: cps-41153, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41153]
-2013-11-06 17:29:22,553-0600 INFO MetaChannel MetaChannel [context: cps-41153, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41153]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:29:22,553-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:29:22,573-0600 INFO RequestHandler Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-06 17:29:22,654-0600 INFO RequestHandler Handler(tag: 42566, SUBMITJOB) unregistering (send)
-Avg stream buf: 0
-2013-11-06 17:29:22,754-0600 INFO Multiplexer Avg stream buf: 0
-2013-11-06 17:29:24,271-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:29:24,271-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:29:24,272-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:29:24,272-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Channel config: null -> {keepalive=null}
-2013-11-06 17:29:24,272-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-Handler(tag: 7431, JOBSTATUS) unregistering (send)
-2013-11-06 17:29:24,272-0600 INFO RequestHandler Handler(tag: 7431, JOBSTATUS) unregistering (send)
-2013-11-06 17:29:24,272-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-TCPChannel [type: server, contact: 127.0.0.1:41151] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:29:24,272-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41151] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-grind.mcs.anl.gov:0 (quality: 50.0) submitting urn:job-2-1383759274663-1383759274664: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:29:24,272-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 50.0) submitting urn:job-2-1383759274663-1383759274664: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-null: Channel shut down
-2013-11-06 17:29:24,273-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-41151, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41151]] binding to NullChannel
-2013-11-06 17:29:24,273-0600 INFO MetaChannel MetaChannel [context: cps-41151, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41151]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41151]
-2013-11-06 17:29:24,273-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41151]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:29:24,273-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:29:24,274-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:29:24,274-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:29:24,274-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:29:24,274-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:29:24,274-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:29:24,274-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:29:24,275-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:29:24,275-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:29:24,275-0600 INFO ChannelManager Channel exception handled
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:29:24,275-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:29:24,275-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41153] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:29:24,275-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41153] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-06 17:29:24,276-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-41153, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41153]] binding to NullChannel
-2013-11-06 17:29:24,276-0600 INFO MetaChannel MetaChannel [context: cps-41153, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41153]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41153]
-2013-11-06 17:29:24,276-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41153]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:29:24,276-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:29:24,276-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:29:24,276-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:29:24,276-0600 INFO ChannelManager Channel exception handled
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:29:24,279-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-Channel is closed. Removing.
-2013-11-06 17:29:24,279-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:29:24,279-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:29:24,279-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41153] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:29:24,279-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41153] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:29:24,280-0600 INFO ChannelManager Channel already shut down
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:29:24,280-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41153]
-2013-11-06 17:29:24,280-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41153]
-2013-11-06 17:29:24,280-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Channel is closed. Removing.
-2013-11-06 17:29:24,280-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:29:24,280-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Handler(tag: 7432, JOBSTATUS) unregistering (send)
-2013-11-06 17:29:24,280-0600 INFO RequestHandler Handler(tag: 7432, JOBSTATUS) unregistering (send)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:29:24,280-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41151] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:29:24,281-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41151] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:29:24,281-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41151]
-2013-11-06 17:29:24,281-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41151]
-2013-11-06 17:29:24,522-0600 DEBUG Cpu Completed
-crush.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:29:24,522-0600 INFO Cpu crush.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:29:24,523-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:29:24,523-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 50, JOBSTATUS) unregistering (send)
-2013-11-06 17:29:24,523-0600 INFO RequestHandler Handler(tag: 50, JOBSTATUS) unregistering (send)
-2013-11-06 17:29:24,523-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-Channel config: null -> {keepalive=null}
-2013-11-06 17:29:24,523-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-06 17:29:24,523-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-TCPChannel [type: server, contact: 127.0.0.1:41152] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:29:24,523-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41152] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-06 17:29:24,524-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-41152, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41152]] binding to NullChannel
-2013-11-06 17:29:24,524-0600 INFO MetaChannel MetaChannel [context: cps-41152, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41152]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41152]
-2013-11-06 17:29:24,524-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41152]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:29:24,524-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:29:24,524-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:29:24,524-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:29:24,524-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:29:24,524-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:29:24,524-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:29:24,524-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:29:24,524-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:29:24,525-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:29:24,525-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 17:29:24,542-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:29:24,542-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:29:24,542-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41152] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:29:24,542-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41152] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:29:24,542-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41152]
-2013-11-06 17:29:24,542-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41152]
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:29:26,291-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:29:26,291-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:29:26,291-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:29:26,291-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-06 17:29:26,291-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7433, JOBSTATUS) unregistering (send)
-2013-11-06 17:29:26,291-0600 INFO RequestHandler Handler(tag: 7433, JOBSTATUS) unregistering (send)
-2013-11-06 17:29:26,291-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:29:32,042-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:29:32,765-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:29:42,772-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:29:52,779-0600 INFO Multiplexer No streams
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172752.273, now=131106-172953.126, channel=TCPChannel [type: server, contact: 127.0.0.1:41148]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:29:53,126-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172752.273, now=131106-172953.126, channel=TCPChannel [type: server, contact: 127.0.0.1:41148]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:29:53,126-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172752.273, now=131106-172953.126, channel=TCPChannel [type: server, contact: 127.0.0.1:41148]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:29:53,126-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172752.273, now=131106-172953.126, channel=TCPChannel [type: server, contact: 127.0.0.1:41148]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:29:53,126-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172752.273, now=131106-172953.126, channel=TCPChannel [type: server, contact: 127.0.0.1:41148]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:29:53,126-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172752.273, now=131106-172953.126, channel=TCPChannel [type: server, contact: 127.0.0.1:41148]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:29:53,126-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172752.273, now=131106-172953.126, channel=TCPChannel [type: server, contact: 127.0.0.1:41148]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:29:53,126-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172752.273, now=131106-172953.126, channel=TCPChannel [type: server, contact: 127.0.0.1:41148]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:29:53,126-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172752.273, now=131106-172953.126, channel=TCPChannel [type: server, contact: 127.0.0.1:41148]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:29:53,127-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172752.273, now=131106-172953.126, channel=TCPChannel [type: server, contact: 127.0.0.1:41148]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:29:53,127-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41148] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172752.273, now=131106-172953.126, channel=TCPChannel [type: server, contact: 127.0.0.1:41148]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:29:53,127-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41148] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172752.273, now=131106-172953.126, channel=TCPChannel [type: server, contact: 127.0.0.1:41148]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:29:53,127-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172752.475, now=131106-172953.332, channel=TCPChannel [type: server, contact: 127.0.0.1:41149]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:29:53,332-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172752.475, now=131106-172953.332, channel=TCPChannel [type: server, contact: 127.0.0.1:41149]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:29:53,332-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172752.475, now=131106-172953.332, channel=TCPChannel [type: server, contact: 127.0.0.1:41149]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:29:53,332-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172752.475, now=131106-172953.332, channel=TCPChannel [type: server, contact: 127.0.0.1:41149]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:29:53,332-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172752.475, now=131106-172953.332, channel=TCPChannel [type: server, contact: 127.0.0.1:41149]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:29:53,332-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172752.475, now=131106-172953.332, channel=TCPChannel [type: server, contact: 127.0.0.1:41149]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:29:53,333-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172752.475, now=131106-172953.332, channel=TCPChannel [type: server, contact: 127.0.0.1:41149]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:29:53,333-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172752.475, now=131106-172953.332, channel=TCPChannel [type: server, contact: 127.0.0.1:41149]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:29:53,333-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172752.475, now=131106-172953.332, channel=TCPChannel [type: server, contact: 127.0.0.1:41149]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:29:53,333-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172752.475, now=131106-172953.332, channel=TCPChannel [type: server, contact: 127.0.0.1:41149]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:29:53,333-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41149] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172752.475, now=131106-172953.332, channel=TCPChannel [type: server, contact: 127.0.0.1:41149]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:29:53,333-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41149] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172752.475, now=131106-172953.332, channel=TCPChannel [type: server, contact: 127.0.0.1:41149]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:29:53,333-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172752.678, now=131106-172953.537, channel=TCPChannel [type: server, contact: 127.0.0.1:41150]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:29:53,537-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172752.678, now=131106-172953.537, channel=TCPChannel [type: server, contact: 127.0.0.1:41150]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:29:53,537-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172752.678, now=131106-172953.537, channel=TCPChannel [type: server, contact: 127.0.0.1:41150]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:29:53,537-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172752.678, now=131106-172953.537, channel=TCPChannel [type: server, contact: 127.0.0.1:41150]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:29:53,538-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41150] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172752.678, now=131106-172953.537, channel=TCPChannel [type: server, contact: 127.0.0.1:41150]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:29:53,538-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41150] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172752.678, now=131106-172953.537, channel=TCPChannel [type: server, contact: 127.0.0.1:41150]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:29:53,538-0600 INFO ChannelManager Channel already shut down
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:29:58,663-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:30:02,786-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:30:08,516-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:30:12,794-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:30:22,802-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:30:32,810-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-06 17:30:35,982-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-06 17:30:42,818-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 17:30:44,990-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 17:30:52,825-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:31:02,833-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:31:12,841-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:31:13,299-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:31:21,464-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:31:22,849-0600 INFO Multiplexer No streams
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172922.230, now=131106-173123.081, channel=TCPChannel [type: server, contact: 127.0.0.1:41151]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:31:23,081-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172922.230, now=131106-173123.081, channel=TCPChannel [type: server, contact: 127.0.0.1:41151]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:31:23,081-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172922.230, now=131106-173123.081, channel=TCPChannel [type: server, contact: 127.0.0.1:41151]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:31:23,081-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172922.230, now=131106-173123.081, channel=TCPChannel [type: server, contact: 127.0.0.1:41151]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:31:23,081-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172922.230, now=131106-173123.081, channel=TCPChannel [type: server, contact: 127.0.0.1:41151]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:31:23,081-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172922.230, now=131106-173123.081, channel=TCPChannel [type: server, contact: 127.0.0.1:41151]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:31:23,081-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172922.230, now=131106-173123.081, channel=TCPChannel [type: server, contact: 127.0.0.1:41151]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:31:23,081-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172922.230, now=131106-173123.081, channel=TCPChannel [type: server, contact: 127.0.0.1:41151]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:31:23,082-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172922.230, now=131106-173123.081, channel=TCPChannel [type: server, contact: 127.0.0.1:41151]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:31:23,082-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172922.230, now=131106-173123.081, channel=TCPChannel [type: server, contact: 127.0.0.1:41151]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:31:23,082-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41151] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172922.230, now=131106-173123.081, channel=TCPChannel [type: server, contact: 127.0.0.1:41151]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:31:23,082-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41151] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172922.230, now=131106-173123.081, channel=TCPChannel [type: server, contact: 127.0.0.1:41151]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:31:23,082-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172922.452, now=131106-173123.288, channel=TCPChannel [type: server, contact: 127.0.0.1:41152]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:31:23,288-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172922.452, now=131106-173123.288, channel=TCPChannel [type: server, contact: 127.0.0.1:41152]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:31:23,288-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172922.452, now=131106-173123.288, channel=TCPChannel [type: server, contact: 127.0.0.1:41152]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:31:23,288-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172922.452, now=131106-173123.288, channel=TCPChannel [type: server, contact: 127.0.0.1:41152]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:31:23,288-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172922.452, now=131106-173123.288, channel=TCPChannel [type: server, contact: 127.0.0.1:41152]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:31:23,288-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172922.452, now=131106-173123.288, channel=TCPChannel [type: server, contact: 127.0.0.1:41152]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:31:23,289-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172922.452, now=131106-173123.288, channel=TCPChannel [type: server, contact: 127.0.0.1:41152]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:31:23,289-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172922.452, now=131106-173123.288, channel=TCPChannel [type: server, contact: 127.0.0.1:41152]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:31:23,289-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172922.452, now=131106-173123.288, channel=TCPChannel [type: server, contact: 127.0.0.1:41152]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:31:23,289-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172922.452, now=131106-173123.288, channel=TCPChannel [type: server, contact: 127.0.0.1:41152]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:31:23,289-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41152] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172922.452, now=131106-173123.288, channel=TCPChannel [type: server, contact: 127.0.0.1:41152]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:31:23,289-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41152] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172922.452, now=131106-173123.288, channel=TCPChannel [type: server, contact: 127.0.0.1:41152]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:31:23,289-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172922.654, now=131106-173123.513, channel=TCPChannel [type: server, contact: 127.0.0.1:41153]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:31:23,513-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172922.654, now=131106-173123.513, channel=TCPChannel [type: server, contact: 127.0.0.1:41153]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:31:23,513-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172922.654, now=131106-173123.513, channel=TCPChannel [type: server, contact: 127.0.0.1:41153]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:31:23,513-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172922.654, now=131106-173123.513, channel=TCPChannel [type: server, contact: 127.0.0.1:41153]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:31:23,514-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41153] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172922.654, now=131106-173123.513, channel=TCPChannel [type: server, contact: 127.0.0.1:41153]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:31:23,514-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41153] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-172922.654, now=131106-173123.513, channel=TCPChannel [type: server, contact: 127.0.0.1:41153]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:31:23,514-0600 INFO ChannelManager Channel already shut down
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:31:30,785-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41154] for addition
-2013-11-06 17:31:30,785-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41154] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41154]
-2013-11-06 17:31:30,785-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41154]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fb7
-2013-11-06 17:31:30,812-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fb7
-MetaChannel [context: cps-41154, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41154]
-2013-11-06 17:31:30,812-0600 INFO MetaChannel MetaChannel [context: cps-41154, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41154]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:31:30,813-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 17:31:30,853-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:31:30,853-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 17:31:30,933-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 17:31:30,934-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 51.0) submitting urn:job-0-1383759274666-1383759274667: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:31:30,934-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 51.0) submitting urn:job-0-1383759274666-1383759274667: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:31:30,935-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 17:31:30,935-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-06 17:31:30,940-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7434, JOBSTATUS) unregistering (send)
-2013-11-06 17:31:30,940-0600 INFO RequestHandler Handler(tag: 7434, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:31:30,992-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41155] for addition
-2013-11-06 17:31:30,992-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41155] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41155]
-2013-11-06 17:31:30,992-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41155]
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7fb6
-2013-11-06 17:31:31,014-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7fb6
-MetaChannel [context: cps-41155, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41155]
-2013-11-06 17:31:31,014-0600 INFO MetaChannel MetaChannel [context: cps-41155, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41155]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:31:31,014-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:31:31,055-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 17:31:31,135-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 17:31:31,135-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-crush.mcs.anl.gov:0 (quality: 10.0) submitting urn:job-1-1383759274669-1383759274670: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:31:31,135-0600 INFO Cpu crush.mcs.anl.gov:0 (quality: 10.0) submitting urn:job-1-1383759274669-1383759274670: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:31:31,142-0600 DEBUG Cpu Active workerid=crush.mcs.anl.gov:000000
-Handler(tag: 51, JOBSTATUS) unregistering (send)
-2013-11-06 17:31:31,142-0600 INFO RequestHandler Handler(tag: 51, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:31:31,197-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41156] for addition
-2013-11-06 17:31:31,197-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41156] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41156]
-2013-11-06 17:31:31,197-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41156]
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7fb5
-2013-11-06 17:31:31,236-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7fb5
-MetaChannel [context: cps-41156, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41156]
-2013-11-06 17:31:31,236-0600 INFO MetaChannel MetaChannel [context: cps-41156, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41156]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:31:31,236-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:31:31,256-0600 INFO RequestHandler Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-06 17:31:31,337-0600 INFO RequestHandler Handler(tag: 42566, SUBMITJOB) unregistering (send)
-Avg stream buf: 0
-2013-11-06 17:31:32,863-0600 INFO Multiplexer Avg stream buf: 0
-2013-11-06 17:31:32,961-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:31:32,961-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:31:32,961-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:31:32,961-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Channel config: null -> {keepalive=null}
-2013-11-06 17:31:32,961-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-Handler(tag: 7435, JOBSTATUS) unregistering (send)
-2013-11-06 17:31:32,962-0600 INFO RequestHandler Handler(tag: 7435, JOBSTATUS) unregistering (send)
-2013-11-06 17:31:32,962-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-TCPChannel [type: server, contact: 127.0.0.1:41154] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:31:32,962-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41154] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-grind.mcs.anl.gov:0 (quality: 52.0) submitting urn:job-2-1383759274672-1383759274673: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:31:32,962-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 52.0) submitting urn:job-2-1383759274672-1383759274673: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-null: Channel shut down
-2013-11-06 17:31:32,962-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-41154, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41154]] binding to NullChannel
-2013-11-06 17:31:32,962-0600 INFO MetaChannel MetaChannel [context: cps-41154, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41154]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41154]
-2013-11-06 17:31:32,963-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41154]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:31:32,963-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:31:32,963-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:31:32,963-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel is closed. Removing.
-2013-11-06 17:31:32,963-0600 INFO Multiplexer Channel is closed. Removing.
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:31:32,963-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:31:32,963-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:31:32,963-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:31:32,963-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:31:32,963-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:31:32,964-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:31:32,964-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:31:32,964-0600 INFO ChannelManager Channel exception handled
-Channel config: null -> {keepalive=null}
-2013-11-06 17:31:32,964-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:31:32,964-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-TCPChannel [type: server, contact: 127.0.0.1:41154] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:31:32,964-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41154] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:31:32,964-0600 INFO ChannelManager Channel already shut down
-Channel config: null -> {keepalive=null}
-2013-11-06 17:31:32,964-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41154]
-2013-11-06 17:31:32,964-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41154]
-TCPChannel [type: server, contact: 127.0.0.1:41156] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:31:32,964-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41156] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-06 17:31:32,965-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-41156, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41156]] binding to NullChannel
-2013-11-06 17:31:32,965-0600 INFO MetaChannel MetaChannel [context: cps-41156, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41156]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41156]
-2013-11-06 17:31:32,965-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41156]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:31:32,965-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:31:32,965-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:31:32,965-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:31:32,965-0600 INFO ChannelManager Channel exception handled
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:31:32,967-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:31:32,969-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:31:32,969-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7436, JOBSTATUS) unregistering (send)
-2013-11-06 17:31:32,969-0600 INFO RequestHandler Handler(tag: 7436, JOBSTATUS) unregistering (send)
-Channel is closed. Removing.
-2013-11-06 17:31:32,984-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:31:32,984-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:31:32,985-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41156] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:31:32,985-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41156] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:31:32,985-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41156]
-2013-11-06 17:31:32,985-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41156]
-2013-11-06 17:31:33,162-0600 DEBUG Cpu Completed
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:31:33,162-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-crush.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:31:33,162-0600 INFO Cpu crush.mcs.anl.gov:0 jobTerminated
-Channel config: null -> {keepalive=null}
-2013-11-06 17:31:33,162-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-06 17:31:33,162-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-TCPChannel [type: server, contact: 127.0.0.1:41155] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:31:33,162-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41155] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 52, JOBSTATUS) unregistering (send)
-2013-11-06 17:31:33,162-0600 INFO RequestHandler Handler(tag: 52, JOBSTATUS) unregistering (send)
-2013-11-06 17:31:33,162-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-null: Channel shut down
-2013-11-06 17:31:33,162-0600 INFO AbstractTCPChannel null: Channel shut down
-2013-11-06 17:31:33,163-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-MetaChannel [context: cps-41155, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41155]] binding to NullChannel
-2013-11-06 17:31:33,163-0600 INFO MetaChannel MetaChannel [context: cps-41155, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41155]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41155]
-2013-11-06 17:31:33,163-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41155]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:31:33,163-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:31:33,163-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:31:33,163-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:31:33,163-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:31:33,163-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:31:33,163-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:31:33,163-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:31:33,164-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:31:33,164-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:31:33,164-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 17:31:33,165-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:31:33,165-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:31:33,166-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41155] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:31:33,166-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41155] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:31:33,166-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41155]
-2013-11-06 17:31:33,166-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41155]
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:31:34,980-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:31:34,981-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:31:34,981-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:31:34,981-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-06 17:31:34,981-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7437, JOBSTATUS) unregistering (send)
-2013-11-06 17:31:34,981-0600 INFO RequestHandler Handler(tag: 7437, JOBSTATUS) unregistering (send)
-2013-11-06 17:31:34,981-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-No streams
-2013-11-06 17:31:42,872-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:31:50,617-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:31:52,880-0600 INFO Multiplexer No streams
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:31:56,772-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41157] for addition
-2013-11-06 17:31:56,773-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41157] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41157]
-2013-11-06 17:31:56,773-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41157]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fb4
-2013-11-06 17:31:56,796-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fb4
-MetaChannel [context: cps-41157, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41157]
-2013-11-06 17:31:56,796-0600 INFO MetaChannel MetaChannel [context: cps-41157, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41157]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:31:56,796-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 17:31:56,837-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:31:56,837-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 17:31:56,917-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 17:31:56,956-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 53.0) submitting urn:job-0-1383759274675-1383759274676: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:31:56,956-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 53.0) submitting urn:job-0-1383759274675-1383759274676: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:31:56,959-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 17:31:56,959-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-06 17:31:56,965-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7438, JOBSTATUS) unregistering (send)
-2013-11-06 17:31:56,965-0600 INFO RequestHandler Handler(tag: 7438, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:31:56,976-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41158] for addition
-2013-11-06 17:31:56,976-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41158] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41158]
-2013-11-06 17:31:56,976-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41158]
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7fb3
-2013-11-06 17:31:56,998-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7fb3
-MetaChannel [context: cps-41158, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41158]
-2013-11-06 17:31:56,998-0600 INFO MetaChannel MetaChannel [context: cps-41158, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41158]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:31:56,998-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:31:57,039-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 17:31:57,119-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 17:31:57,159-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-crush.mcs.anl.gov:0 (quality: 11.0) submitting urn:job-1-1383759274678-1383759274679: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:31:57,159-0600 INFO Cpu crush.mcs.anl.gov:0 (quality: 11.0) submitting urn:job-1-1383759274678-1383759274679: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:31:57,167-0600 DEBUG Cpu Active workerid=crush.mcs.anl.gov:000000
-Handler(tag: 53, JOBSTATUS) unregistering (send)
-2013-11-06 17:31:57,167-0600 INFO RequestHandler Handler(tag: 53, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:31:57,179-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41159] for addition
-2013-11-06 17:31:57,180-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41159] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41159]
-2013-11-06 17:31:57,180-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41159]
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7fb2
-2013-11-06 17:31:57,220-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7fb2
-MetaChannel [context: cps-41159, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41159]
-2013-11-06 17:31:57,220-0600 INFO MetaChannel MetaChannel [context: cps-41159, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41159]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:31:57,220-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:31:57,241-0600 INFO RequestHandler Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-06 17:31:57,321-0600 INFO RequestHandler Handler(tag: 42566, SUBMITJOB) unregistering (send)
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:31:57,938-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:31:58,977-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:31:58,977-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:31:58,977-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:31:58,977-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:31:58,977-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-Handler(tag: 7439, JOBSTATUS) unregistering (send)
-2013-11-06 17:31:58,977-0600 INFO RequestHandler Handler(tag: 7439, JOBSTATUS) unregistering (send)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:31:58,978-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-grind.mcs.anl.gov:0 (quality: 54.0) submitting urn:job-2-1383759274681-1383759274682: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:31:58,978-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 54.0) submitting urn:job-2-1383759274681-1383759274682: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-TCPChannel [type: server, contact: 127.0.0.1:41157] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:31:58,978-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41157] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-06 17:31:58,979-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-41157, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41157]] binding to NullChannel
-2013-11-06 17:31:58,979-0600 INFO MetaChannel MetaChannel [context: cps-41157, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41157]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41157]
-2013-11-06 17:31:58,979-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41157]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:31:58,979-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:31:58,979-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:31:58,979-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:31:58,979-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:31:58,979-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:31:58,980-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:31:58,980-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:31:58,980-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:31:58,980-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:31:58,980-0600 INFO ChannelManager Channel exception handled
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:31:58,980-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:31:58,980-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41159] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:31:58,980-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41159] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-06 17:31:58,980-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-41159, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41159]] binding to NullChannel
-2013-11-06 17:31:58,981-0600 INFO MetaChannel MetaChannel [context: cps-41159, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41159]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41159]
-2013-11-06 17:31:58,981-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41159]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:31:58,981-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:31:58,981-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:31:58,981-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:31:58,981-0600 INFO ChannelManager Channel exception handled
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:31:58,984-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:31:58,985-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:31:58,985-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7440, JOBSTATUS) unregistering (send)
-2013-11-06 17:31:58,985-0600 INFO RequestHandler Handler(tag: 7440, JOBSTATUS) unregistering (send)
-Channel is closed. Removing.
-2013-11-06 17:31:58,986-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:31:58,986-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:31:58,986-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41157] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:31:58,986-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41157] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:31:58,987-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41157]
-2013-11-06 17:31:58,987-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41157]
-Channel is closed. Removing.
-2013-11-06 17:31:58,987-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:31:58,987-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:31:58,987-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41159] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:31:58,987-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41159] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:31:58,987-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41159]
-2013-11-06 17:31:58,987-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41159]
-2013-11-06 17:31:59,186-0600 DEBUG Cpu Completed
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:31:59,186-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-crush.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:31:59,186-0600 INFO Cpu crush.mcs.anl.gov:0 jobTerminated
-Channel config: null -> {keepalive=null}
-2013-11-06 17:31:59,186-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-06 17:31:59,186-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-TCPChannel [type: server, contact: 127.0.0.1:41158] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:31:59,186-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41158] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 54, JOBSTATUS) unregistering (send)
-2013-11-06 17:31:59,187-0600 INFO RequestHandler Handler(tag: 54, JOBSTATUS) unregistering (send)
-2013-11-06 17:31:59,187-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-null: Channel shut down
-2013-11-06 17:31:59,187-0600 INFO AbstractTCPChannel null: Channel shut down
-2013-11-06 17:31:59,187-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-MetaChannel [context: cps-41158, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41158]] binding to NullChannel
-2013-11-06 17:31:59,187-0600 INFO MetaChannel MetaChannel [context: cps-41158, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41158]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41158]
-2013-11-06 17:31:59,187-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41158]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:31:59,187-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:31:59,187-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:31:59,187-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:31:59,188-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:31:59,188-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel is closed. Removing.
-2013-11-06 17:31:59,188-0600 INFO Multiplexer Channel is closed. Removing.
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:31:59,188-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:31:59,188-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:31:59,188-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:31:59,188-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:31:59,188-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:31:59,188-0600 INFO ChannelManager Channel exception handled
-Channel config: null -> {keepalive=null}
-2013-11-06 17:31:59,188-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41158] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:31:59,188-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41158] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:31:59,189-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41158]
-2013-11-06 17:31:59,189-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41158]
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:32:01,012-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:32:01,013-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:32:01,013-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:32:01,013-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-06 17:32:01,013-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7441, JOBSTATUS) unregistering (send)
-2013-11-06 17:32:01,013-0600 INFO RequestHandler Handler(tag: 7441, JOBSTATUS) unregistering (send)
-2013-11-06 17:32:01,013-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-No streams
-2013-11-06 17:32:02,899-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:32:12,907-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:32:22,916-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:32:27,935-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:32:32,924-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-06 17:32:34,411-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-No streams
-2013-11-06 17:32:42,932-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:32:52,940-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:33:02,948-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:33:05,253-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:33:10,886-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:33:12,956-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:33:22,964-0600 INFO Multiplexer No streams
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173130.933, now=131106-173331.786, channel=TCPChannel [type: server, contact: 127.0.0.1:41154]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:33:31,786-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173130.933, now=131106-173331.786, channel=TCPChannel [type: server, contact: 127.0.0.1:41154]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:33:31,786-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173130.933, now=131106-173331.786, channel=TCPChannel [type: server, contact: 127.0.0.1:41154]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:33:31,786-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173130.933, now=131106-173331.786, channel=TCPChannel [type: server, contact: 127.0.0.1:41154]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:33:31,786-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173130.933, now=131106-173331.786, channel=TCPChannel [type: server, contact: 127.0.0.1:41154]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:33:31,786-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173130.933, now=131106-173331.786, channel=TCPChannel [type: server, contact: 127.0.0.1:41154]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:33:31,786-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173130.933, now=131106-173331.786, channel=TCPChannel [type: server, contact: 127.0.0.1:41154]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:33:31,786-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173130.933, now=131106-173331.786, channel=TCPChannel [type: server, contact: 127.0.0.1:41154]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:33:31,787-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173130.933, now=131106-173331.786, channel=TCPChannel [type: server, contact: 127.0.0.1:41154]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:33:31,787-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173130.933, now=131106-173331.786, channel=TCPChannel [type: server, contact: 127.0.0.1:41154]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:33:31,787-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41154] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173130.933, now=131106-173331.786, channel=TCPChannel [type: server, contact: 127.0.0.1:41154]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:33:31,787-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41154] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173130.933, now=131106-173331.786, channel=TCPChannel [type: server, contact: 127.0.0.1:41154]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:33:31,787-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173131.135, now=131106-173331.992, channel=TCPChannel [type: server, contact: 127.0.0.1:41155]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:33:31,992-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173131.135, now=131106-173331.992, channel=TCPChannel [type: server, contact: 127.0.0.1:41155]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:33:31,992-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173131.135, now=131106-173331.992, channel=TCPChannel [type: server, contact: 127.0.0.1:41155]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:33:31,992-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173131.135, now=131106-173331.992, channel=TCPChannel [type: server, contact: 127.0.0.1:41155]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:33:31,992-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173131.135, now=131106-173331.992, channel=TCPChannel [type: server, contact: 127.0.0.1:41155]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:33:31,992-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173131.135, now=131106-173331.992, channel=TCPChannel [type: server, contact: 127.0.0.1:41155]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:33:31,993-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173131.135, now=131106-173331.992, channel=TCPChannel [type: server, contact: 127.0.0.1:41155]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:33:31,993-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173131.135, now=131106-173331.992, channel=TCPChannel [type: server, contact: 127.0.0.1:41155]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:33:31,993-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173131.135, now=131106-173331.992, channel=TCPChannel [type: server, contact: 127.0.0.1:41155]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:33:31,993-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173131.135, now=131106-173331.992, channel=TCPChannel [type: server, contact: 127.0.0.1:41155]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:33:31,993-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41155] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173131.135, now=131106-173331.992, channel=TCPChannel [type: server, contact: 127.0.0.1:41155]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:33:31,993-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41155] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173131.135, now=131106-173331.992, channel=TCPChannel [type: server, contact: 127.0.0.1:41155]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:33:31,993-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173131.337, now=131106-173332.197, channel=TCPChannel [type: server, contact: 127.0.0.1:41156]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:33:32,197-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173131.337, now=131106-173332.197, channel=TCPChannel [type: server, contact: 127.0.0.1:41156]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:33:32,198-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173131.337, now=131106-173332.197, channel=TCPChannel [type: server, contact: 127.0.0.1:41156]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:33:32,198-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173131.337, now=131106-173332.197, channel=TCPChannel [type: server, contact: 127.0.0.1:41156]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:33:32,198-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41156] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173131.337, now=131106-173332.197, channel=TCPChannel [type: server, contact: 127.0.0.1:41156]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:33:32,198-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41156] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173131.337, now=131106-173332.197, channel=TCPChannel [type: server, contact: 127.0.0.1:41156]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:33:32,198-0600 INFO ChannelManager Channel already shut down
-No streams
-2013-11-06 17:33:32,973-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:33:42,571-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:33:42,981-0600 INFO Multiplexer No streams
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:33:43,659-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41165] for addition
-2013-11-06 17:33:43,659-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41165] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41165]
-2013-11-06 17:33:43,659-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41165]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fb1
-2013-11-06 17:33:43,684-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fb1
-MetaChannel [context: cps-41165, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41165]
-2013-11-06 17:33:43,684-0600 INFO MetaChannel MetaChannel [context: cps-41165, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41165]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:33:43,684-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:33:43,724-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 17:33:43,725-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 17:33:43,805-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 17:33:43,829-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 55.0) submitting urn:job-0-1383759274684-1383759274685: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:33:43,829-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 55.0) submitting urn:job-0-1383759274684-1383759274685: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:33:43,830-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 17:33:43,830-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-06 17:33:43,836-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7442, JOBSTATUS) unregistering (send)
-2013-11-06 17:33:43,836-0600 INFO RequestHandler Handler(tag: 7442, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:33:43,864-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41166] for addition
-2013-11-06 17:33:43,864-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41166] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41166]
-2013-11-06 17:33:43,864-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41166]
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7fb0
-2013-11-06 17:33:43,886-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7fb0
-MetaChannel [context: cps-41166, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41166]
-2013-11-06 17:33:43,886-0600 INFO MetaChannel MetaChannel [context: cps-41166, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41166]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:33:43,886-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:33:43,947-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 17:33:44,027-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 17:33:44,030-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-crush.mcs.anl.gov:0 (quality: 12.0) submitting urn:job-1-1383759274687-1383759274688: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:33:44,030-0600 INFO Cpu crush.mcs.anl.gov:0 (quality: 12.0) submitting urn:job-1-1383759274687-1383759274688: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:33:44,037-0600 DEBUG Cpu Active workerid=crush.mcs.anl.gov:000000
-Handler(tag: 55, JOBSTATUS) unregistering (send)
-2013-11-06 17:33:44,037-0600 INFO RequestHandler Handler(tag: 55, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:33:44,088-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41167] for addition
-2013-11-06 17:33:44,088-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41167] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41167]
-2013-11-06 17:33:44,088-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41167]
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7faf
-2013-11-06 17:33:44,108-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7faf
-MetaChannel [context: cps-41167, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41167]
-2013-11-06 17:33:44,108-0600 INFO MetaChannel MetaChannel [context: cps-41167, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41167]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:33:44,108-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:33:44,149-0600 INFO RequestHandler Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-06 17:33:44,229-0600 INFO RequestHandler Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-06 17:33:45,868-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:33:45,868-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:33:45,868-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:33:45,868-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 7443, JOBSTATUS) unregistering (send)
-2013-11-06 17:33:45,868-0600 INFO RequestHandler Handler(tag: 7443, JOBSTATUS) unregistering (send)
-2013-11-06 17:33:45,868-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-Channel config: null -> {keepalive=null}
-2013-11-06 17:33:45,868-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41165] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:33:45,869-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41165] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-grind.mcs.anl.gov:0 (quality: 56.0) submitting urn:job-2-1383759274690-1383759274691: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:33:45,869-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 56.0) submitting urn:job-2-1383759274690-1383759274691: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-null: Channel shut down
-2013-11-06 17:33:45,869-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-41165, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41165]] binding to NullChannel
-2013-11-06 17:33:45,869-0600 INFO MetaChannel MetaChannel [context: cps-41165, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41165]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41165]
-2013-11-06 17:33:45,869-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41165]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:33:45,869-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:33:45,869-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:33:45,869-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:33:45,869-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:33:45,870-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:33:45,870-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:33:45,870-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:33:45,870-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:33:45,870-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:33:45,870-0600 INFO ChannelManager Channel exception handled
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:33:45,871-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:33:45,871-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41167] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:33:45,871-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41167] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-06 17:33:45,871-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-41167, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41167]] binding to NullChannel
-2013-11-06 17:33:45,871-0600 INFO MetaChannel MetaChannel [context: cps-41167, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41167]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41167]
-2013-11-06 17:33:45,871-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41167]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:33:45,871-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:33:45,871-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:33:45,871-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:33:45,871-0600 INFO ChannelManager Channel exception handled
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:33:45,874-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-Channel is closed. Removing.
-2013-11-06 17:33:45,874-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:33:45,874-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:33:45,875-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:33:45,875-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Channel config: null -> {keepalive=null}
-2013-11-06 17:33:45,875-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-Handler(tag: 7444, JOBSTATUS) unregistering (send)
-2013-11-06 17:33:45,875-0600 INFO RequestHandler Handler(tag: 7444, JOBSTATUS) unregistering (send)
-TCPChannel [type: server, contact: 127.0.0.1:41167] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:33:45,875-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41167] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:33:45,875-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41167]
-2013-11-06 17:33:45,875-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41167]
-Channel is closed. Removing.
-2013-11-06 17:33:45,875-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:33:45,875-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:33:45,875-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41165] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:33:45,876-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41165] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:33:45,876-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41165]
-2013-11-06 17:33:45,876-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41165]
-2013-11-06 17:33:46,099-0600 DEBUG Cpu Completed
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:33:46,099-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-crush.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:33:46,099-0600 INFO Cpu crush.mcs.anl.gov:0 jobTerminated
-Channel config: null -> {keepalive=null}
-2013-11-06 17:33:46,099-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-06 17:33:46,099-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-TCPChannel [type: server, contact: 127.0.0.1:41166] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:33:46,099-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41166] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 56, JOBSTATUS) unregistering (send)
-2013-11-06 17:33:46,099-0600 INFO RequestHandler Handler(tag: 56, JOBSTATUS) unregistering (send)
-2013-11-06 17:33:46,100-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-null: Channel shut down
-2013-11-06 17:33:46,100-0600 INFO AbstractTCPChannel null: Channel shut down
-2013-11-06 17:33:46,100-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-MetaChannel [context: cps-41166, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41166]] binding to NullChannel
-2013-11-06 17:33:46,100-0600 INFO MetaChannel MetaChannel [context: cps-41166, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41166]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41166]
-2013-11-06 17:33:46,100-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41166]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:33:46,100-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:33:46,100-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:33:46,100-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:33:46,100-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:33:46,101-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:33:46,101-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:33:46,101-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:33:46,101-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:33:46,101-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:33:46,101-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 17:33:46,116-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:33:46,116-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:33:46,117-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41166] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:33:46,117-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41166] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:33:46,117-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41166]
-2013-11-06 17:33:46,117-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41166]
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:33:47,360-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:33:47,886-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:33:47,886-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:33:47,886-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:33:47,887-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-06 17:33:47,887-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-2013-11-06 17:33:47,887-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-Handler(tag: 7445, JOBSTATUS) unregistering (send)
-2013-11-06 17:33:47,887-0600 INFO RequestHandler Handler(tag: 7445, JOBSTATUS) unregistering (send)
-No streams
-2013-11-06 17:33:52,996-0600 INFO Multiplexer No streams
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173156.917, now=131106-173357.772, channel=TCPChannel [type: server, contact: 127.0.0.1:41157]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:33:57,772-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173156.917, now=131106-173357.772, channel=TCPChannel [type: server, contact: 127.0.0.1:41157]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:33:57,773-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173156.917, now=131106-173357.772, channel=TCPChannel [type: server, contact: 127.0.0.1:41157]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:33:57,773-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173156.917, now=131106-173357.772, channel=TCPChannel [type: server, contact: 127.0.0.1:41157]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:33:57,773-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173156.917, now=131106-173357.772, channel=TCPChannel [type: server, contact: 127.0.0.1:41157]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:33:57,773-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173156.917, now=131106-173357.772, channel=TCPChannel [type: server, contact: 127.0.0.1:41157]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:33:57,773-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173156.917, now=131106-173357.772, channel=TCPChannel [type: server, contact: 127.0.0.1:41157]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:33:57,773-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173156.917, now=131106-173357.772, channel=TCPChannel [type: server, contact: 127.0.0.1:41157]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:33:57,773-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173156.917, now=131106-173357.772, channel=TCPChannel [type: server, contact: 127.0.0.1:41157]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:33:57,773-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173156.917, now=131106-173357.772, channel=TCPChannel [type: server, contact: 127.0.0.1:41157]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:33:57,773-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41157] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173156.917, now=131106-173357.772, channel=TCPChannel [type: server, contact: 127.0.0.1:41157]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:33:57,773-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41157] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173156.917, now=131106-173357.772, channel=TCPChannel [type: server, contact: 127.0.0.1:41157]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:33:57,773-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173157.119, now=131106-173357.977, channel=TCPChannel [type: server, contact: 127.0.0.1:41158]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:33:57,977-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173157.119, now=131106-173357.977, channel=TCPChannel [type: server, contact: 127.0.0.1:41158]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:33:57,977-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173157.119, now=131106-173357.977, channel=TCPChannel [type: server, contact: 127.0.0.1:41158]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:33:57,977-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173157.119, now=131106-173357.977, channel=TCPChannel [type: server, contact: 127.0.0.1:41158]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:33:57,977-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173157.119, now=131106-173357.977, channel=TCPChannel [type: server, contact: 127.0.0.1:41158]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:33:57,977-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173157.119, now=131106-173357.977, channel=TCPChannel [type: server, contact: 127.0.0.1:41158]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:33:57,977-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173157.119, now=131106-173357.977, channel=TCPChannel [type: server, contact: 127.0.0.1:41158]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:33:57,977-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173157.119, now=131106-173357.977, channel=TCPChannel [type: server, contact: 127.0.0.1:41158]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:33:57,977-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173157.119, now=131106-173357.977, channel=TCPChannel [type: server, contact: 127.0.0.1:41158]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:33:57,977-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173157.119, now=131106-173357.977, channel=TCPChannel [type: server, contact: 127.0.0.1:41158]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:33:57,977-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41158] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173157.119, now=131106-173357.977, channel=TCPChannel [type: server, contact: 127.0.0.1:41158]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:33:57,977-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41158] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173157.119, now=131106-173357.977, channel=TCPChannel [type: server, contact: 127.0.0.1:41158]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:33:57,977-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173157.321, now=131106-173358.179, channel=TCPChannel [type: server, contact: 127.0.0.1:41159]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:33:58,179-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173157.321, now=131106-173358.179, channel=TCPChannel [type: server, contact: 127.0.0.1:41159]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:33:58,179-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173157.321, now=131106-173358.179, channel=TCPChannel [type: server, contact: 127.0.0.1:41159]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:33:58,179-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173157.321, now=131106-173358.179, channel=TCPChannel [type: server, contact: 127.0.0.1:41159]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:33:58,179-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41159] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173157.321, now=131106-173358.179, channel=TCPChannel [type: server, contact: 127.0.0.1:41159]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:33:58,179-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41159] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173157.321, now=131106-173358.179, channel=TCPChannel [type: server, contact: 127.0.0.1:41159]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:33:58,179-0600 INFO ChannelManager Channel already shut down
-No streams
-2013-11-06 17:34:03,004-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:34:13,013-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-06 17:34:19,889-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-06 17:34:23,021-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:34:23,834-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:34:33,029-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:34:43,037-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:34:53,045-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:34:57,207-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:35:00,308-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:35:03,053-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:35:13,061-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:35:23,069-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:35:33,078-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:35:34,525-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 17:35:36,783-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 17:35:43,086-0600 INFO Multiplexer No streams
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173343.805, now=131106-173544.659, channel=TCPChannel [type: server, contact: 127.0.0.1:41165]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:35:44,659-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173343.805, now=131106-173544.659, channel=TCPChannel [type: server, contact: 127.0.0.1:41165]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:35:44,660-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173343.805, now=131106-173544.659, channel=TCPChannel [type: server, contact: 127.0.0.1:41165]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:35:44,660-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173343.805, now=131106-173544.659, channel=TCPChannel [type: server, contact: 127.0.0.1:41165]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:35:44,660-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173343.805, now=131106-173544.659, channel=TCPChannel [type: server, contact: 127.0.0.1:41165]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:35:44,660-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173343.805, now=131106-173544.659, channel=TCPChannel [type: server, contact: 127.0.0.1:41165]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:35:44,660-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173343.805, now=131106-173544.659, channel=TCPChannel [type: server, contact: 127.0.0.1:41165]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:35:44,660-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173343.805, now=131106-173544.659, channel=TCPChannel [type: server, contact: 127.0.0.1:41165]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:35:44,660-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173343.805, now=131106-173544.659, channel=TCPChannel [type: server, contact: 127.0.0.1:41165]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:35:44,660-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173343.805, now=131106-173544.659, channel=TCPChannel [type: server, contact: 127.0.0.1:41165]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:35:44,660-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41165] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173343.805, now=131106-173544.659, channel=TCPChannel [type: server, contact: 127.0.0.1:41165]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:35:44,660-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41165] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173343.805, now=131106-173544.659, channel=TCPChannel [type: server, contact: 127.0.0.1:41165]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:35:44,660-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173344.027, now=131106-173544.865, channel=TCPChannel [type: server, contact: 127.0.0.1:41166]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:35:44,865-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173344.027, now=131106-173544.865, channel=TCPChannel [type: server, contact: 127.0.0.1:41166]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:35:44,865-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173344.027, now=131106-173544.865, channel=TCPChannel [type: server, contact: 127.0.0.1:41166]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:35:44,865-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173344.027, now=131106-173544.865, channel=TCPChannel [type: server, contact: 127.0.0.1:41166]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:35:44,865-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173344.027, now=131106-173544.865, channel=TCPChannel [type: server, contact: 127.0.0.1:41166]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:35:44,865-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173344.027, now=131106-173544.865, channel=TCPChannel [type: server, contact: 127.0.0.1:41166]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:35:44,865-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173344.027, now=131106-173544.865, channel=TCPChannel [type: server, contact: 127.0.0.1:41166]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:35:44,865-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173344.027, now=131106-173544.865, channel=TCPChannel [type: server, contact: 127.0.0.1:41166]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:35:44,865-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173344.027, now=131106-173544.865, channel=TCPChannel [type: server, contact: 127.0.0.1:41166]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:35:44,865-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173344.027, now=131106-173544.865, channel=TCPChannel [type: server, contact: 127.0.0.1:41166]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:35:44,865-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41166] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173344.027, now=131106-173544.865, channel=TCPChannel [type: server, contact: 127.0.0.1:41166]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:35:44,865-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41166] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173344.027, now=131106-173544.865, channel=TCPChannel [type: server, contact: 127.0.0.1:41166]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:35:44,865-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173344.229, now=131106-173545.088, channel=TCPChannel [type: server, contact: 127.0.0.1:41167]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:35:45,088-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173344.229, now=131106-173545.088, channel=TCPChannel [type: server, contact: 127.0.0.1:41167]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:35:45,088-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173344.229, now=131106-173545.088, channel=TCPChannel [type: server, contact: 127.0.0.1:41167]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:35:45,088-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173344.229, now=131106-173545.088, channel=TCPChannel [type: server, contact: 127.0.0.1:41167]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:35:45,088-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41167] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173344.229, now=131106-173545.088, channel=TCPChannel [type: server, contact: 127.0.0.1:41167]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:35:45,088-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41167] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173344.229, now=131106-173545.088, channel=TCPChannel [type: server, contact: 127.0.0.1:41167]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:35:45,088-0600 INFO ChannelManager Channel already shut down
-No streams
-2013-11-06 17:35:53,094-0600 INFO Multiplexer No streams
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:35:54,601-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41168] for addition
-2013-11-06 17:35:54,601-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41168] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41168]
-2013-11-06 17:35:54,602-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41168]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fae
-2013-11-06 17:35:54,640-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fae
-MetaChannel [context: cps-41168, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41168]
-2013-11-06 17:35:54,640-0600 INFO MetaChannel MetaChannel [context: cps-41168, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41168]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:35:54,640-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:35:54,661-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 17:35:54,661-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 17:35:54,741-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 17:35:54,781-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 57.0) submitting urn:job-0-1383759274693-1383759274694: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:35:54,781-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 57.0) submitting urn:job-0-1383759274693-1383759274694: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:35:54,782-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 17:35:54,782-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-06 17:35:54,789-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7446, JOBSTATUS) unregistering (send)
-2013-11-06 17:35:54,789-0600 INFO RequestHandler Handler(tag: 7446, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:35:54,801-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41169] for addition
-2013-11-06 17:35:54,801-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41169] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41169]
-2013-11-06 17:35:54,801-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41169]
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7fad
-2013-11-06 17:35:54,822-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7fad
-MetaChannel [context: cps-41169, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41169]
-2013-11-06 17:35:54,822-0600 INFO MetaChannel MetaChannel [context: cps-41169, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41169]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:35:54,822-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:35:54,863-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 17:35:54,943-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 17:35:54,982-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-crush.mcs.anl.gov:0 (quality: 13.0) submitting urn:job-1-1383759274696-1383759274697: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:35:54,982-0600 INFO Cpu crush.mcs.anl.gov:0 (quality: 13.0) submitting urn:job-1-1383759274696-1383759274697: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:35:54,989-0600 DEBUG Cpu Active workerid=crush.mcs.anl.gov:000000
-Handler(tag: 57, JOBSTATUS) unregistering (send)
-2013-11-06 17:35:54,989-0600 INFO RequestHandler Handler(tag: 57, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:35:55,007-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41170] for addition
-2013-11-06 17:35:55,008-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41170] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41170]
-2013-11-06 17:35:55,008-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41170]
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7fac
-2013-11-06 17:35:55,044-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7fac
-MetaChannel [context: cps-41170, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41170]
-2013-11-06 17:35:55,044-0600 INFO MetaChannel MetaChannel [context: cps-41170, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41170]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:35:55,044-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:35:55,085-0600 INFO RequestHandler Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-06 17:35:55,165-0600 INFO RequestHandler Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-06 17:35:56,810-0600 DEBUG Cpu Completed
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:35:56,810-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:35:56,810-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-Channel config: null -> {keepalive=null}
-2013-11-06 17:35:56,810-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-06 17:35:56,810-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-TCPChannel [type: server, contact: 127.0.0.1:41168] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:35:56,810-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41168] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 7447, JOBSTATUS) unregistering (send)
-2013-11-06 17:35:56,810-0600 INFO RequestHandler Handler(tag: 7447, JOBSTATUS) unregistering (send)
-null: Channel shut down
-2013-11-06 17:35:56,811-0600 INFO AbstractTCPChannel null: Channel shut down
-2013-11-06 17:35:56,810-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-MetaChannel [context: cps-41168, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41168]] binding to NullChannel
-2013-11-06 17:35:56,811-0600 INFO MetaChannel MetaChannel [context: cps-41168, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41168]] binding to NullChannel
-grind.mcs.anl.gov:0 (quality: 58.0) submitting urn:job-2-1383759274699-1383759274700: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:35:56,811-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 58.0) submitting urn:job-2-1383759274699-1383759274700: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41168]
-2013-11-06 17:35:56,811-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41168]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:35:56,811-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:35:56,811-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:35:56,812-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:35:56,812-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:35:56,812-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:35:56,812-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:35:56,812-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:35:56,812-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:35:56,812-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:35:56,812-0600 INFO ChannelManager Channel exception handled
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:35:56,812-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:35:56,813-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41170] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:35:56,813-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41170] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-06 17:35:56,813-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-41170, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41170]] binding to NullChannel
-2013-11-06 17:35:56,813-0600 INFO MetaChannel MetaChannel [context: cps-41170, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41170]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41170]
-2013-11-06 17:35:56,813-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41170]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:35:56,813-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:35:56,813-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:35:56,813-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:35:56,813-0600 INFO ChannelManager Channel exception handled
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:35:56,815-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:35:56,816-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:35:56,816-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7448, JOBSTATUS) unregistering (send)
-2013-11-06 17:35:56,816-0600 INFO RequestHandler Handler(tag: 7448, JOBSTATUS) unregistering (send)
-Channel is closed. Removing.
-2013-11-06 17:35:56,830-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:35:56,830-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:35:56,830-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41170] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:35:56,830-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41170] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:35:56,831-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41170]
-2013-11-06 17:35:56,831-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41170]
-Channel is closed. Removing.
-2013-11-06 17:35:56,831-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:35:56,831-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:35:56,831-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41168] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:35:56,831-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41168] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:35:56,831-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41168]
-2013-11-06 17:35:56,831-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41168]
-2013-11-06 17:35:57,029-0600 DEBUG Cpu Completed
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:35:57,029-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-crush.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:35:57,029-0600 INFO Cpu crush.mcs.anl.gov:0 jobTerminated
-Channel config: null -> {keepalive=null}
-2013-11-06 17:35:57,029-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-06 17:35:57,030-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-TCPChannel [type: server, contact: 127.0.0.1:41169] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:35:57,030-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41169] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:35:57,030-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-null: Channel shut down
-2013-11-06 17:35:57,030-0600 INFO AbstractTCPChannel null: Channel shut down
-Handler(tag: 58, JOBSTATUS) unregistering (send)
-2013-11-06 17:35:57,030-0600 INFO RequestHandler Handler(tag: 58, JOBSTATUS) unregistering (send)
-MetaChannel [context: cps-41169, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41169]] binding to NullChannel
-2013-11-06 17:35:57,030-0600 INFO MetaChannel MetaChannel [context: cps-41169, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41169]] binding to NullChannel
-2013-11-06 17:35:57,030-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41169]
-2013-11-06 17:35:57,030-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41169]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:35:57,030-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:35:57,031-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:35:57,031-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:35:57,031-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:35:57,031-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:35:57,031-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:35:57,031-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:35:57,031-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:35:57,031-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:35:57,032-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 17:35:57,032-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:35:57,032-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:35:57,032-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41169] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:35:57,032-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41169] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:35:57,032-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41169]
-2013-11-06 17:35:57,032-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41169]
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:35:58,839-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:35:58,839-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:35:58,839-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:35:58,839-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-06 17:35:58,839-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7449, JOBSTATUS) unregistering (send)
-2013-11-06 17:35:58,839-0600 INFO RequestHandler Handler(tag: 7449, JOBSTATUS) unregistering (send)
-2013-11-06 17:35:58,840-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-No streams
-2013-11-06 17:36:03,109-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-06 17:36:11,843-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-No streams
-2013-11-06 17:36:13,117-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:36:13,256-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:36:23,125-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:36:33,133-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:36:43,140-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-06 17:36:49,162-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:36:49,730-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:36:53,148-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:37:03,156-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:37:13,164-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:37:23,172-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:37:26,204-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:37:26,480-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:37:33,180-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:37:43,187-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:37:53,195-0600 INFO Multiplexer No streams
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173554.741, now=131106-173755.601, channel=TCPChannel [type: server, contact: 127.0.0.1:41168]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:37:55,601-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173554.741, now=131106-173755.601, channel=TCPChannel [type: server, contact: 127.0.0.1:41168]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:37:55,601-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173554.741, now=131106-173755.601, channel=TCPChannel [type: server, contact: 127.0.0.1:41168]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:37:55,601-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173554.741, now=131106-173755.601, channel=TCPChannel [type: server, contact: 127.0.0.1:41168]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:37:55,601-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173554.741, now=131106-173755.601, channel=TCPChannel [type: server, contact: 127.0.0.1:41168]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:37:55,601-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173554.741, now=131106-173755.601, channel=TCPChannel [type: server, contact: 127.0.0.1:41168]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:37:55,601-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173554.741, now=131106-173755.601, channel=TCPChannel [type: server, contact: 127.0.0.1:41168]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:37:55,601-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173554.741, now=131106-173755.601, channel=TCPChannel [type: server, contact: 127.0.0.1:41168]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:37:55,602-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173554.741, now=131106-173755.601, channel=TCPChannel [type: server, contact: 127.0.0.1:41168]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:37:55,602-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173554.741, now=131106-173755.601, channel=TCPChannel [type: server, contact: 127.0.0.1:41168]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:37:55,602-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41168] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173554.741, now=131106-173755.601, channel=TCPChannel [type: server, contact: 127.0.0.1:41168]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:37:55,602-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41168] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173554.741, now=131106-173755.601, channel=TCPChannel [type: server, contact: 127.0.0.1:41168]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:37:55,602-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173554.943, now=131106-173755.801, channel=TCPChannel [type: server, contact: 127.0.0.1:41169]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:37:55,801-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173554.943, now=131106-173755.801, channel=TCPChannel [type: server, contact: 127.0.0.1:41169]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:37:55,801-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173554.943, now=131106-173755.801, channel=TCPChannel [type: server, contact: 127.0.0.1:41169]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:37:55,801-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173554.943, now=131106-173755.801, channel=TCPChannel [type: server, contact: 127.0.0.1:41169]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:37:55,801-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173554.943, now=131106-173755.801, channel=TCPChannel [type: server, contact: 127.0.0.1:41169]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:37:55,801-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173554.943, now=131106-173755.801, channel=TCPChannel [type: server, contact: 127.0.0.1:41169]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:37:55,801-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173554.943, now=131106-173755.801, channel=TCPChannel [type: server, contact: 127.0.0.1:41169]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:37:55,801-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173554.943, now=131106-173755.801, channel=TCPChannel [type: server, contact: 127.0.0.1:41169]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:37:55,802-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173554.943, now=131106-173755.801, channel=TCPChannel [type: server, contact: 127.0.0.1:41169]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:37:55,802-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173554.943, now=131106-173755.801, channel=TCPChannel [type: server, contact: 127.0.0.1:41169]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:37:55,802-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41169] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173554.943, now=131106-173755.801, channel=TCPChannel [type: server, contact: 127.0.0.1:41169]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:37:55,802-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41169] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173554.943, now=131106-173755.801, channel=TCPChannel [type: server, contact: 127.0.0.1:41169]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:37:55,802-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173555.165, now=131106-173756.007, channel=TCPChannel [type: server, contact: 127.0.0.1:41170]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:37:56,007-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173555.165, now=131106-173756.007, channel=TCPChannel [type: server, contact: 127.0.0.1:41170]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:37:56,007-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173555.165, now=131106-173756.007, channel=TCPChannel [type: server, contact: 127.0.0.1:41170]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:37:56,007-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173555.165, now=131106-173756.007, channel=TCPChannel [type: server, contact: 127.0.0.1:41170]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:37:56,007-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41170] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173555.165, now=131106-173756.007, channel=TCPChannel [type: server, contact: 127.0.0.1:41170]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:37:56,007-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41170] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173555.165, now=131106-173756.007, channel=TCPChannel [type: server, contact: 127.0.0.1:41170]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:37:56,008-0600 INFO ChannelManager Channel already shut down
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 17:38:02,679-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 17:38:03,203-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:38:03,798-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:38:13,212-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:38:23,220-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:38:33,228-0600 INFO Multiplexer No streams
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:38:34,281-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41171] for addition
-2013-11-06 17:38:34,281-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41171] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41171]
-2013-11-06 17:38:34,281-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41171]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fab
-2013-11-06 17:38:34,312-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fab
-MetaChannel [context: cps-41171, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41171]
-2013-11-06 17:38:34,312-0600 INFO MetaChannel MetaChannel [context: cps-41171, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41171]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:38:34,312-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:38:34,353-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 17:38:34,353-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 17:38:34,433-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 17:38:34,466-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 59.0) submitting urn:job-0-1383759274702-1383759274703: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:38:34,466-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 59.0) submitting urn:job-0-1383759274702-1383759274703: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:38:34,466-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 17:38:34,466-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-06 17:38:34,471-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7450, JOBSTATUS) unregistering (send)
-2013-11-06 17:38:34,471-0600 INFO RequestHandler Handler(tag: 7450, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:38:34,492-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41172] for addition
-2013-11-06 17:38:34,492-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41172] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41172]
-2013-11-06 17:38:34,492-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41172]
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7faa
-2013-11-06 17:38:34,514-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7faa
-MetaChannel [context: cps-41172, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41172]
-2013-11-06 17:38:34,514-0600 INFO MetaChannel MetaChannel [context: cps-41172, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41172]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:38:34,514-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:38:34,575-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 17:38:34,655-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 17:38:34,667-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-crush.mcs.anl.gov:0 (quality: 14.0) submitting urn:job-1-1383759274705-1383759274706: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:38:34,667-0600 INFO Cpu crush.mcs.anl.gov:0 (quality: 14.0) submitting urn:job-1-1383759274705-1383759274706: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:38:34,681-0600 DEBUG Cpu Active workerid=crush.mcs.anl.gov:000000
-Handler(tag: 59, JOBSTATUS) unregistering (send)
-2013-11-06 17:38:34,681-0600 INFO RequestHandler Handler(tag: 59, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:38:34,716-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41173] for addition
-2013-11-06 17:38:34,716-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41173] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41173]
-2013-11-06 17:38:34,716-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41173]
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7fa9
-2013-11-06 17:38:34,756-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7fa9
-MetaChannel [context: cps-41173, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41173]
-2013-11-06 17:38:34,756-0600 INFO MetaChannel MetaChannel [context: cps-41173, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41173]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:38:34,756-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:38:34,776-0600 INFO RequestHandler Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-06 17:38:34,859-0600 INFO RequestHandler Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-06 17:38:36,488-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:38:36,488-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:38:36,488-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:38:36,488-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 7451, JOBSTATUS) unregistering (send)
-2013-11-06 17:38:36,488-0600 INFO RequestHandler Handler(tag: 7451, JOBSTATUS) unregistering (send)
-2013-11-06 17:38:36,488-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-Channel config: null -> {keepalive=null}
-2013-11-06 17:38:36,489-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41171] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:38:36,489-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41171] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-grind.mcs.anl.gov:0 (quality: 60.0) submitting urn:job-2-1383759274708-1383759274709: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:38:36,489-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 60.0) submitting urn:job-2-1383759274708-1383759274709: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-null: Channel shut down
-2013-11-06 17:38:36,489-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-41171, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41171]] binding to NullChannel
-2013-11-06 17:38:36,490-0600 INFO MetaChannel MetaChannel [context: cps-41171, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41171]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41171]
-2013-11-06 17:38:36,490-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41171]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:38:36,490-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:38:36,490-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:38:36,490-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:38:36,490-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:38:36,490-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:38:36,490-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:38:36,490-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:38:36,490-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:38:36,491-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:38:36,491-0600 INFO ChannelManager Channel exception handled
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:38:36,491-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:38:36,491-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41173] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:38:36,491-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41173] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-06 17:38:36,491-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-41173, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41173]] binding to NullChannel
-2013-11-06 17:38:36,491-0600 INFO MetaChannel MetaChannel [context: cps-41173, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41173]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41173]
-2013-11-06 17:38:36,491-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41173]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:38:36,491-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:38:36,491-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:38:36,492-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:38:36,492-0600 INFO ChannelManager Channel exception handled
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:38:36,494-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:38:36,495-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:38:36,495-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7452, JOBSTATUS) unregistering (send)
-2013-11-06 17:38:36,495-0600 INFO RequestHandler Handler(tag: 7452, JOBSTATUS) unregistering (send)
-Channel is closed. Removing.
-2013-11-06 17:38:36,504-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:38:36,504-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:38:36,504-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41171] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:38:36,504-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41171] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:38:36,504-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41171]
-2013-11-06 17:38:36,504-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41171]
-Channel is closed. Removing.
-2013-11-06 17:38:36,504-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:38:36,504-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:38:36,504-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41173] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:38:36,504-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41173] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:38:36,504-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41173]
-2013-11-06 17:38:36,504-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41173]
-2013-11-06 17:38:36,701-0600 DEBUG Cpu Completed
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:38:36,701-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-crush.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:38:36,701-0600 INFO Cpu crush.mcs.anl.gov:0 jobTerminated
-Channel config: null -> {keepalive=null}
-2013-11-06 17:38:36,701-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-06 17:38:36,701-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-TCPChannel [type: server, contact: 127.0.0.1:41172] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:38:36,701-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41172] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 60, JOBSTATUS) unregistering (send)
-2013-11-06 17:38:36,701-0600 INFO RequestHandler Handler(tag: 60, JOBSTATUS) unregistering (send)
-2013-11-06 17:38:36,701-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-null: Channel shut down
-2013-11-06 17:38:36,701-0600 INFO AbstractTCPChannel null: Channel shut down
-2013-11-06 17:38:36,702-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-MetaChannel [context: cps-41172, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41172]] binding to NullChannel
-2013-11-06 17:38:36,702-0600 INFO MetaChannel MetaChannel [context: cps-41172, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41172]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41172]
-2013-11-06 17:38:36,702-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41172]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:38:36,702-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:38:36,702-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:38:36,702-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:38:36,702-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:38:36,702-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:38:36,702-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:38:36,702-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:38:36,703-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:38:36,703-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:38:36,703-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 17:38:36,705-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:38:36,705-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:38:36,705-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41172] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:38:36,705-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41172] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:38:36,705-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41172]
-2013-11-06 17:38:36,705-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41172]
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:38:38,514-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:38:38,514-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:38:38,515-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:38:38,515-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-06 17:38:38,515-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7453, JOBSTATUS) unregistering (send)
-2013-11-06 17:38:38,515-0600 INFO RequestHandler Handler(tag: 7453, JOBSTATUS) unregistering (send)
-2013-11-06 17:38:38,515-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:38:39,152-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-06 17:38:41,115-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-No streams
-2013-11-06 17:38:43,244-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:38:53,252-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:39:03,261-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:39:13,269-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:39:15,626-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:39:18,434-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:39:23,277-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:39:33,285-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:39:43,293-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:39:52,100-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:39:53,301-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:39:55,752-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:40:03,310-0600 INFO Multiplexer No streams
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:40:07,123-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41174] for addition
-2013-11-06 17:40:07,123-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41174] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41174]
-2013-11-06 17:40:07,123-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41174]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fa8
-2013-11-06 17:40:07,161-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fa8
-MetaChannel [context: cps-41174, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41174]
-2013-11-06 17:40:07,162-0600 INFO MetaChannel MetaChannel [context: cps-41174, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41174]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:40:07,162-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:40:07,202-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 17:40:07,202-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 17:40:07,283-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 17:40:07,314-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 61.0) submitting urn:job-0-1383759274711-1383759274712: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:40:07,314-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 61.0) submitting urn:job-0-1383759274711-1383759274712: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:40:07,315-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 17:40:07,315-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-06 17:40:07,320-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7454, JOBSTATUS) unregistering (send)
-2013-11-06 17:40:07,320-0600 INFO RequestHandler Handler(tag: 7454, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:40:07,344-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41175] for addition
-2013-11-06 17:40:07,344-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41175] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41175]
-2013-11-06 17:40:07,344-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41175]
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7fa7
-2013-11-06 17:40:07,384-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7fa7
-MetaChannel [context: cps-41175, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41175]
-2013-11-06 17:40:07,384-0600 INFO MetaChannel MetaChannel [context: cps-41175, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41175]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:40:07,384-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:40:07,404-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 17:40:07,485-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 17:40:07,515-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-crush.mcs.anl.gov:0 (quality: 15.0) submitting urn:job-1-1383759274714-1383759274715: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:40:07,515-0600 INFO Cpu crush.mcs.anl.gov:0 (quality: 15.0) submitting urn:job-1-1383759274714-1383759274715: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:40:07,522-0600 DEBUG Cpu Active workerid=crush.mcs.anl.gov:000000
-Handler(tag: 61, JOBSTATUS) unregistering (send)
-2013-11-06 17:40:07,522-0600 INFO RequestHandler Handler(tag: 61, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:40:07,545-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41176] for addition
-2013-11-06 17:40:07,545-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41176] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41176]
-2013-11-06 17:40:07,545-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41176]
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7fa6
-2013-11-06 17:40:07,565-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7fa6
-MetaChannel [context: cps-41176, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41176]
-2013-11-06 17:40:07,565-0600 INFO MetaChannel MetaChannel [context: cps-41176, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41176]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:40:07,566-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:40:07,606-0600 INFO RequestHandler Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-06 17:40:07,686-0600 INFO RequestHandler Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-06 17:40:09,339-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:40:09,339-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:40:09,339-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:40:09,339-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 7455, JOBSTATUS) unregistering (send)
-2013-11-06 17:40:09,339-0600 INFO RequestHandler Handler(tag: 7455, JOBSTATUS) unregistering (send)
-2013-11-06 17:40:09,339-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-Channel config: null -> {keepalive=null}
-2013-11-06 17:40:09,340-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-grind.mcs.anl.gov:0 (quality: 62.0) submitting urn:job-2-1383759274717-1383759274718: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:40:09,340-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 62.0) submitting urn:job-2-1383759274717-1383759274718: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-TCPChannel [type: server, contact: 127.0.0.1:41174] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:40:09,340-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41174] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-06 17:40:09,340-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-41174, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41174]] binding to NullChannel
-2013-11-06 17:40:09,341-0600 INFO MetaChannel MetaChannel [context: cps-41174, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41174]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41174]
-2013-11-06 17:40:09,341-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41174]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:40:09,341-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:40:09,341-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:40:09,341-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:40:09,341-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:40:09,341-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:40:09,341-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:40:09,341-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:40:09,341-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:40:09,342-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:40:09,342-0600 INFO ChannelManager Channel exception handled
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:40:09,342-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:40:09,342-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41176] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:40:09,342-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41176] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-06 17:40:09,342-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-41176, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41176]] binding to NullChannel
-2013-11-06 17:40:09,342-0600 INFO MetaChannel MetaChannel [context: cps-41176, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41176]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41176]
-2013-11-06 17:40:09,342-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41176]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:40:09,342-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:40:09,342-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:40:09,342-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:40:09,343-0600 INFO ChannelManager Channel exception handled
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:40:09,345-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:40:09,346-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:40:09,346-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7456, JOBSTATUS) unregistering (send)
-2013-11-06 17:40:09,346-0600 INFO RequestHandler Handler(tag: 7456, JOBSTATUS) unregistering (send)
-Channel is closed. Removing.
-2013-11-06 17:40:09,351-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:40:09,351-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:40:09,352-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41174] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:40:09,352-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41174] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:40:09,352-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41174]
-2013-11-06 17:40:09,352-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41174]
-Channel is closed. Removing.
-2013-11-06 17:40:09,352-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:40:09,352-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:40:09,352-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41176] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:40:09,352-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41176] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:40:09,352-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41176]
-2013-11-06 17:40:09,352-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41176]
-2013-11-06 17:40:09,544-0600 DEBUG Cpu Completed
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:40:09,544-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-crush.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:40:09,544-0600 INFO Cpu crush.mcs.anl.gov:0 jobTerminated
-Channel config: null -> {keepalive=null}
-2013-11-06 17:40:09,544-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-06 17:40:09,544-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-TCPChannel [type: server, contact: 127.0.0.1:41175] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:40:09,544-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41175] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 62, JOBSTATUS) unregistering (send)
-2013-11-06 17:40:09,544-0600 INFO RequestHandler Handler(tag: 62, JOBSTATUS) unregistering (send)
-2013-11-06 17:40:09,544-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-null: Channel shut down
-2013-11-06 17:40:09,544-0600 INFO AbstractTCPChannel null: Channel shut down
-2013-11-06 17:40:09,545-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-MetaChannel [context: cps-41175, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41175]] binding to NullChannel
-2013-11-06 17:40:09,545-0600 INFO MetaChannel MetaChannel [context: cps-41175, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41175]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41175]
-2013-11-06 17:40:09,545-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41175]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:40:09,545-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:40:09,545-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:40:09,545-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:40:09,545-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:40:09,545-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:40:09,545-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:40:09,545-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:40:09,545-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:40:09,545-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:40:09,546-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 17:40:09,553-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:40:09,553-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:40:09,553-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41175] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:40:09,553-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41175] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:40:09,553-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41175]
-2013-11-06 17:40:09,553-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41175]
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:40:11,357-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:40:11,357-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:40:11,357-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:40:11,357-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-06 17:40:11,357-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7457, JOBSTATUS) unregistering (send)
-2013-11-06 17:40:11,357-0600 INFO RequestHandler Handler(tag: 7457, JOBSTATUS) unregistering (send)
-2013-11-06 17:40:11,357-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-No streams
-2013-11-06 17:40:13,324-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:40:23,332-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-06 17:40:28,575-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:40:33,070-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:40:33,340-0600 INFO Multiplexer No streams
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173834.433, now=131106-174035.281, channel=TCPChannel [type: server, contact: 127.0.0.1:41171]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:40:35,281-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173834.433, now=131106-174035.281, channel=TCPChannel [type: server, contact: 127.0.0.1:41171]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:40:35,282-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173834.433, now=131106-174035.281, channel=TCPChannel [type: server, contact: 127.0.0.1:41171]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:40:35,282-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173834.433, now=131106-174035.281, channel=TCPChannel [type: server, contact: 127.0.0.1:41171]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:40:35,282-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173834.433, now=131106-174035.281, channel=TCPChannel [type: server, contact: 127.0.0.1:41171]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:40:35,282-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173834.433, now=131106-174035.281, channel=TCPChannel [type: server, contact: 127.0.0.1:41171]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:40:35,282-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173834.433, now=131106-174035.281, channel=TCPChannel [type: server, contact: 127.0.0.1:41171]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:40:35,282-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173834.433, now=131106-174035.281, channel=TCPChannel [type: server, contact: 127.0.0.1:41171]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:40:35,282-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173834.433, now=131106-174035.281, channel=TCPChannel [type: server, contact: 127.0.0.1:41171]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:40:35,282-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173834.433, now=131106-174035.281, channel=TCPChannel [type: server, contact: 127.0.0.1:41171]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:40:35,282-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41171] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173834.433, now=131106-174035.281, channel=TCPChannel [type: server, contact: 127.0.0.1:41171]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:40:35,282-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41171] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173834.433, now=131106-174035.281, channel=TCPChannel [type: server, contact: 127.0.0.1:41171]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:40:35,282-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173834.655, now=131106-174035.493, channel=TCPChannel [type: server, contact: 127.0.0.1:41172]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:40:35,493-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173834.655, now=131106-174035.493, channel=TCPChannel [type: server, contact: 127.0.0.1:41172]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:40:35,493-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173834.655, now=131106-174035.493, channel=TCPChannel [type: server, contact: 127.0.0.1:41172]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:40:35,493-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173834.655, now=131106-174035.493, channel=TCPChannel [type: server, contact: 127.0.0.1:41172]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:40:35,493-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173834.655, now=131106-174035.493, channel=TCPChannel [type: server, contact: 127.0.0.1:41172]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:40:35,493-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173834.655, now=131106-174035.493, channel=TCPChannel [type: server, contact: 127.0.0.1:41172]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:40:35,493-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173834.655, now=131106-174035.493, channel=TCPChannel [type: server, contact: 127.0.0.1:41172]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:40:35,493-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173834.655, now=131106-174035.493, channel=TCPChannel [type: server, contact: 127.0.0.1:41172]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:40:35,493-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173834.655, now=131106-174035.493, channel=TCPChannel [type: server, contact: 127.0.0.1:41172]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:40:35,493-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173834.655, now=131106-174035.493, channel=TCPChannel [type: server, contact: 127.0.0.1:41172]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:40:35,493-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41172] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173834.655, now=131106-174035.493, channel=TCPChannel [type: server, contact: 127.0.0.1:41172]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:40:35,493-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41172] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173834.655, now=131106-174035.493, channel=TCPChannel [type: server, contact: 127.0.0.1:41172]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:40:35,493-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173834.859, now=131106-174035.716, channel=TCPChannel [type: server, contact: 127.0.0.1:41173]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:40:35,716-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173834.859, now=131106-174035.716, channel=TCPChannel [type: server, contact: 127.0.0.1:41173]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:40:35,716-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173834.859, now=131106-174035.716, channel=TCPChannel [type: server, contact: 127.0.0.1:41173]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:40:35,716-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173834.859, now=131106-174035.716, channel=TCPChannel [type: server, contact: 127.0.0.1:41173]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:40:35,716-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41173] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173834.859, now=131106-174035.716, channel=TCPChannel [type: server, contact: 127.0.0.1:41173]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:40:35,716-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41173] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-173834.859, now=131106-174035.716, channel=TCPChannel [type: server, contact: 127.0.0.1:41173]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:40:35,716-0600 INFO ChannelManager Channel already shut down
-No streams
-2013-11-06 17:40:43,348-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:40:53,357-0600 INFO Multiplexer No streams
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:40:59,585-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41177] for addition
-2013-11-06 17:40:59,585-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41177] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41177]
-2013-11-06 17:40:59,585-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41177]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fa5
-2013-11-06 17:40:59,615-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fa5
-MetaChannel [context: cps-41177, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41177]
-2013-11-06 17:40:59,615-0600 INFO MetaChannel MetaChannel [context: cps-41177, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41177]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:40:59,615-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:40:59,656-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 17:40:59,656-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 17:40:59,736-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 17:40:59,762-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 63.0) submitting urn:job-0-1383759274720-1383759274721: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:40:59,763-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 63.0) submitting urn:job-0-1383759274720-1383759274721: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:40:59,763-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 17:40:59,763-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-06 17:40:59,770-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7458, JOBSTATUS) unregistering (send)
-2013-11-06 17:40:59,770-0600 INFO RequestHandler Handler(tag: 7458, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:40:59,796-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41178] for addition
-2013-11-06 17:40:59,796-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41178] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41178]
-2013-11-06 17:40:59,796-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41178]
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7fa4
-2013-11-06 17:40:59,817-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7fa4
-MetaChannel [context: cps-41178, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41178]
-2013-11-06 17:40:59,817-0600 INFO MetaChannel MetaChannel [context: cps-41178, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41178]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:40:59,817-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:40:59,858-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 17:40:59,938-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 17:40:59,964-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-crush.mcs.anl.gov:0 (quality: 16.0) submitting urn:job-1-1383759274723-1383759274724: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:40:59,964-0600 INFO Cpu crush.mcs.anl.gov:0 (quality: 16.0) submitting urn:job-1-1383759274723-1383759274724: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:40:59,970-0600 DEBUG Cpu Active workerid=crush.mcs.anl.gov:000000
-Handler(tag: 63, JOBSTATUS) unregistering (send)
-2013-11-06 17:40:59,970-0600 INFO RequestHandler Handler(tag: 63, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:40:59,996-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41179] for addition
-2013-11-06 17:40:59,996-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41179] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41179]
-2013-11-06 17:40:59,996-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41179]
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7fa3
-2013-11-06 17:41:00,019-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7fa3
-MetaChannel [context: cps-41179, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41179]
-2013-11-06 17:41:00,019-0600 INFO MetaChannel MetaChannel [context: cps-41179, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41179]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:41:00,019-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:41:00,059-0600 INFO RequestHandler Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-06 17:41:00,140-0600 INFO RequestHandler Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-06 17:41:01,781-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:41:01,781-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:41:01,781-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:41:01,781-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 7459, JOBSTATUS) unregistering (send)
-2013-11-06 17:41:01,781-0600 INFO RequestHandler Handler(tag: 7459, JOBSTATUS) unregistering (send)
-2013-11-06 17:41:01,781-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-Channel config: null -> {keepalive=null}
-2013-11-06 17:41:01,781-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-grind.mcs.anl.gov:0 (quality: 64.0) submitting urn:job-2-1383759274726-1383759274727: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:41:01,781-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 64.0) submitting urn:job-2-1383759274726-1383759274727: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-TCPChannel [type: server, contact: 127.0.0.1:41177] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:41:01,782-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41177] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-06 17:41:01,782-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-41177, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41177]] binding to NullChannel
-2013-11-06 17:41:01,782-0600 INFO MetaChannel MetaChannel [context: cps-41177, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41177]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41177]
-2013-11-06 17:41:01,782-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41177]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:41:01,782-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:41:01,782-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:41:01,783-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:41:01,783-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:41:01,783-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:41:01,783-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:41:01,783-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:41:01,783-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:41:01,783-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:41:01,783-0600 INFO ChannelManager Channel exception handled
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:41:01,783-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:41:01,784-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41179] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:41:01,784-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41179] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-06 17:41:01,784-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-41179, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41179]] binding to NullChannel
-2013-11-06 17:41:01,784-0600 INFO MetaChannel MetaChannel [context: cps-41179, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41179]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41179]
-2013-11-06 17:41:01,784-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41179]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:41:01,784-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:41:01,784-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:41:01,784-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:41:01,784-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 17:41:01,785-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:41:01,785-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:41:01,785-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41177] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:41:01,785-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41177] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:41:01,785-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41177]
-2013-11-06 17:41:01,785-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41177]
-Channel is closed. Removing.
-2013-11-06 17:41:01,785-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:41:01,786-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:41:01,786-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41179] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:41:01,786-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41179] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:41:01,786-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41179]
-2013-11-06 17:41:01,786-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41179]
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:41:01,787-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:41:01,788-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:41:01,788-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7460, JOBSTATUS) unregistering (send)
-2013-11-06 17:41:01,788-0600 INFO RequestHandler Handler(tag: 7460, JOBSTATUS) unregistering (send)
-2013-11-06 17:41:01,982-0600 DEBUG Cpu Completed
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:41:01,982-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-crush.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:41:01,982-0600 INFO Cpu crush.mcs.anl.gov:0 jobTerminated
-Channel config: null -> {keepalive=null}
-2013-11-06 17:41:01,982-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-06 17:41:01,982-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-TCPChannel [type: server, contact: 127.0.0.1:41178] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:41:01,982-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41178] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 64, JOBSTATUS) unregistering (send)
-2013-11-06 17:41:01,982-0600 INFO RequestHandler Handler(tag: 64, JOBSTATUS) unregistering (send)
-2013-11-06 17:41:01,982-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-null: Channel shut down
-2013-11-06 17:41:01,982-0600 INFO AbstractTCPChannel null: Channel shut down
-2013-11-06 17:41:01,982-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-MetaChannel [context: cps-41178, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41178]] binding to NullChannel
-2013-11-06 17:41:01,982-0600 INFO MetaChannel MetaChannel [context: cps-41178, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41178]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41178]
-2013-11-06 17:41:01,982-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41178]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:41:01,983-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:41:01,983-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:41:01,983-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:41:01,983-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:41:01,983-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:41:01,983-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:41:01,983-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:41:01,983-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:41:01,983-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:41:01,983-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 17:41:01,986-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:41:01,987-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:41:01,987-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41178] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:41:01,987-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41178] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:41:01,987-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41178]
-2013-11-06 17:41:01,987-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41178]
-No streams
-2013-11-06 17:41:03,371-0600 INFO Multiplexer No streams
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:41:03,800-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:41:03,800-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:41:03,800-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:41:03,800-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-06 17:41:03,800-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7461, JOBSTATUS) unregistering (send)
-2013-11-06 17:41:03,800-0600 INFO RequestHandler Handler(tag: 7461, JOBSTATUS) unregistering (send)
-2013-11-06 17:41:03,800-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-06 17:41:05,047-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:41:10,388-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:41:13,379-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:41:23,387-0600 INFO Multiplexer No streams
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:41:26,723-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41180] for addition
-2013-11-06 17:41:26,723-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41180] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41180]
-2013-11-06 17:41:26,723-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41180]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fa2
-2013-11-06 17:41:26,757-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7fa2
-MetaChannel [context: cps-41180, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41180]
-2013-11-06 17:41:26,757-0600 INFO MetaChannel MetaChannel [context: cps-41180, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41180]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:41:26,757-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:41:26,798-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 17:41:26,798-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 17:41:26,878-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 17:41:26,926-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 65.0) submitting urn:job-0-1383759274729-1383759274730: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:41:26,926-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 65.0) submitting urn:job-0-1383759274729-1383759274730: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:41:26,928-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 17:41:26,928-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-06 17:41:26,933-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7462, JOBSTATUS) unregistering (send)
-2013-11-06 17:41:26,933-0600 INFO RequestHandler Handler(tag: 7462, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:41:26,936-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41181] for addition
-2013-11-06 17:41:26,936-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41181] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41181]
-2013-11-06 17:41:26,936-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41181]
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7fa1
-2013-11-06 17:41:26,959-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7fa1
-MetaChannel [context: cps-41181, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41181]
-2013-11-06 17:41:26,959-0600 INFO MetaChannel MetaChannel [context: cps-41181, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41181]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:41:26,959-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:41:26,999-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 17:41:27,080-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 17:41:27,128-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-crush.mcs.anl.gov:0 (quality: 17.0) submitting urn:job-1-1383759274732-1383759274733: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:41:27,128-0600 INFO Cpu crush.mcs.anl.gov:0 (quality: 17.0) submitting urn:job-1-1383759274732-1383759274733: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:41:27,136-0600 DEBUG Cpu Active workerid=crush.mcs.anl.gov:000000
-Handler(tag: 65, JOBSTATUS) unregistering (send)
-2013-11-06 17:41:27,136-0600 INFO RequestHandler Handler(tag: 65, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:41:27,140-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41182] for addition
-2013-11-06 17:41:27,140-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41182] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41182]
-2013-11-06 17:41:27,140-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41182]
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7fa0
-2013-11-06 17:41:27,160-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7fa0
-MetaChannel [context: cps-41182, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41182]
-2013-11-06 17:41:27,160-0600 INFO MetaChannel MetaChannel [context: cps-41182, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41182]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:41:27,160-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:41:27,201-0600 INFO RequestHandler Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-06 17:41:27,281-0600 INFO RequestHandler Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-06 17:41:29,037-0600 DEBUG Cpu Completed
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:41:29,037-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:41:29,037-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-Channel config: null -> {keepalive=null}
-2013-11-06 17:41:29,037-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-06 17:41:29,037-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-TCPChannel [type: server, contact: 127.0.0.1:41180] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:41:29,037-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41180] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 7463, JOBSTATUS) unregistering (send)
-2013-11-06 17:41:29,037-0600 INFO RequestHandler Handler(tag: 7463, JOBSTATUS) unregistering (send)
-2013-11-06 17:41:29,037-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-null: Channel shut down
-2013-11-06 17:41:29,038-0600 INFO AbstractTCPChannel null: Channel shut down
-grind.mcs.anl.gov:0 (quality: 66.0) submitting urn:job-2-1383759274735-1383759274736: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:41:29,038-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 66.0) submitting urn:job-2-1383759274735-1383759274736: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-MetaChannel [context: cps-41180, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41180]] binding to NullChannel
-2013-11-06 17:41:29,038-0600 INFO MetaChannel MetaChannel [context: cps-41180, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41180]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41180]
-2013-11-06 17:41:29,039-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41180]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:41:29,039-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:41:29,039-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:41:29,039-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:41:29,039-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:41:29,039-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:41:29,040-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:41:29,040-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:41:29,040-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:41:29,040-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:41:29,040-0600 INFO ChannelManager Channel exception handled
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:41:29,040-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:41:29,040-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41182] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:41:29,040-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41182] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-06 17:41:29,040-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-41182, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41182]] binding to NullChannel
-2013-11-06 17:41:29,040-0600 INFO MetaChannel MetaChannel [context: cps-41182, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41182]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41182]
-2013-11-06 17:41:29,040-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41182]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:41:29,041-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:41:29,041-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:41:29,041-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:41:29,041-0600 INFO ChannelManager Channel exception handled
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:41:29,044-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:41:29,044-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:41:29,045-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7464, JOBSTATUS) unregistering (send)
-2013-11-06 17:41:29,045-0600 INFO RequestHandler Handler(tag: 7464, JOBSTATUS) unregistering (send)
-Channel is closed. Removing.
-2013-11-06 17:41:29,047-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:41:29,047-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:41:29,047-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41180] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:41:29,047-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41180] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:41:29,047-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41180]
-2013-11-06 17:41:29,047-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41180]
-Channel is closed. Removing.
-2013-11-06 17:41:29,047-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:41:29,047-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:41:29,047-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41182] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:41:29,047-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41182] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:41:29,047-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41182]
-2013-11-06 17:41:29,047-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41182]
-2013-11-06 17:41:29,154-0600 DEBUG Cpu Completed
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:41:29,154-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-crush.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:41:29,155-0600 INFO Cpu crush.mcs.anl.gov:0 jobTerminated
-Channel config: null -> {keepalive=null}
-2013-11-06 17:41:29,155-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-06 17:41:29,155-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-TCPChannel [type: server, contact: 127.0.0.1:41181] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:41:29,155-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41181] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 66, JOBSTATUS) unregistering (send)
-2013-11-06 17:41:29,155-0600 INFO RequestHandler Handler(tag: 66, JOBSTATUS) unregistering (send)
-2013-11-06 17:41:29,155-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-null: Channel shut down
-2013-11-06 17:41:29,155-0600 INFO AbstractTCPChannel null: Channel shut down
-2013-11-06 17:41:29,155-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-MetaChannel [context: cps-41181, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41181]] binding to NullChannel
-2013-11-06 17:41:29,155-0600 INFO MetaChannel MetaChannel [context: cps-41181, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41181]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41181]
-2013-11-06 17:41:29,156-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41181]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:41:29,156-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:41:29,156-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:41:29,156-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:41:29,156-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:41:29,156-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:41:29,156-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:41:29,156-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:41:29,156-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:41:29,156-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:41:29,156-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 17:41:29,168-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:41:29,168-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:41:29,168-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41181] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:41:29,168-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41181] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:41:29,168-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41181]
-2013-11-06 17:41:29,168-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41181]
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:41:31,055-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:41:31,055-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:41:31,055-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:41:31,055-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-06 17:41:31,055-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7465, JOBSTATUS) unregistering (send)
-2013-11-06 17:41:31,055-0600 INFO RequestHandler Handler(tag: 7465, JOBSTATUS) unregistering (send)
-2013-11-06 17:41:31,055-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-No streams
-2013-11-06 17:41:33,400-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:41:41,522-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:41:43,408-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-06 17:41:47,706-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-06 17:41:53,416-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:42:03,424-0600 INFO Multiplexer No streams
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174007.283, now=131106-174208.123, channel=TCPChannel [type: server, contact: 127.0.0.1:41174]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:42:08,123-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174007.283, now=131106-174208.123, channel=TCPChannel [type: server, contact: 127.0.0.1:41174]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:42:08,123-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174007.283, now=131106-174208.123, channel=TCPChannel [type: server, contact: 127.0.0.1:41174]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:42:08,123-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174007.283, now=131106-174208.123, channel=TCPChannel [type: server, contact: 127.0.0.1:41174]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:42:08,123-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174007.283, now=131106-174208.123, channel=TCPChannel [type: server, contact: 127.0.0.1:41174]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:42:08,123-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174007.283, now=131106-174208.123, channel=TCPChannel [type: server, contact: 127.0.0.1:41174]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:42:08,123-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174007.283, now=131106-174208.123, channel=TCPChannel [type: server, contact: 127.0.0.1:41174]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:42:08,123-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174007.283, now=131106-174208.123, channel=TCPChannel [type: server, contact: 127.0.0.1:41174]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:42:08,123-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174007.283, now=131106-174208.123, channel=TCPChannel [type: server, contact: 127.0.0.1:41174]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:42:08,124-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174007.283, now=131106-174208.123, channel=TCPChannel [type: server, contact: 127.0.0.1:41174]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:42:08,124-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41174] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174007.283, now=131106-174208.123, channel=TCPChannel [type: server, contact: 127.0.0.1:41174]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:42:08,124-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41174] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174007.283, now=131106-174208.123, channel=TCPChannel [type: server, contact: 127.0.0.1:41174]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:42:08,124-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174007.484, now=131106-174208.344, channel=TCPChannel [type: server, contact: 127.0.0.1:41175]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:42:08,344-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174007.484, now=131106-174208.344, channel=TCPChannel [type: server, contact: 127.0.0.1:41175]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:42:08,344-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174007.484, now=131106-174208.344, channel=TCPChannel [type: server, contact: 127.0.0.1:41175]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:42:08,344-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174007.484, now=131106-174208.344, channel=TCPChannel [type: server, contact: 127.0.0.1:41175]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:42:08,344-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174007.484, now=131106-174208.344, channel=TCPChannel [type: server, contact: 127.0.0.1:41175]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:42:08,344-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174007.484, now=131106-174208.344, channel=TCPChannel [type: server, contact: 127.0.0.1:41175]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:42:08,344-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174007.484, now=131106-174208.344, channel=TCPChannel [type: server, contact: 127.0.0.1:41175]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:42:08,344-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174007.484, now=131106-174208.344, channel=TCPChannel [type: server, contact: 127.0.0.1:41175]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:42:08,345-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174007.484, now=131106-174208.344, channel=TCPChannel [type: server, contact: 127.0.0.1:41175]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:42:08,345-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174007.484, now=131106-174208.344, channel=TCPChannel [type: server, contact: 127.0.0.1:41175]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:42:08,345-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41175] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174007.484, now=131106-174208.344, channel=TCPChannel [type: server, contact: 127.0.0.1:41175]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:42:08,345-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41175] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174007.484, now=131106-174208.344, channel=TCPChannel [type: server, contact: 127.0.0.1:41175]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:42:08,345-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174007.686, now=131106-174208.545, channel=TCPChannel [type: server, contact: 127.0.0.1:41176]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:42:08,545-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174007.686, now=131106-174208.545, channel=TCPChannel [type: server, contact: 127.0.0.1:41176]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:42:08,545-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174007.686, now=131106-174208.545, channel=TCPChannel [type: server, contact: 127.0.0.1:41176]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:42:08,545-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174007.686, now=131106-174208.545, channel=TCPChannel [type: server, contact: 127.0.0.1:41176]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:42:08,545-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41176] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174007.686, now=131106-174208.545, channel=TCPChannel [type: server, contact: 127.0.0.1:41176]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:42:08,545-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41176] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174007.686, now=131106-174208.545, channel=TCPChannel [type: server, contact: 127.0.0.1:41176]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:42:08,545-0600 INFO ChannelManager Channel already shut down
-No streams
-2013-11-06 17:42:13,432-0600 INFO Multiplexer No streams
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:42:16,988-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41184] for addition
-2013-11-06 17:42:16,988-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41184] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41184]
-2013-11-06 17:42:16,988-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41184]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f9f
-2013-11-06 17:42:17,023-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f9f
-MetaChannel [context: cps-41184, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41184]
-2013-11-06 17:42:17,023-0600 INFO MetaChannel MetaChannel [context: cps-41184, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41184]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:42:17,024-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:42:17,064-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 17:42:17,064-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 17:42:17,144-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 17:42:17,157-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 67.0) submitting urn:job-0-1383759274738-1383759274739: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:42:17,157-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 67.0) submitting urn:job-0-1383759274738-1383759274739: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:42:17,158-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 17:42:17,158-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-06 17:42:17,164-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7466, JOBSTATUS) unregistering (send)
-2013-11-06 17:42:17,165-0600 INFO RequestHandler Handler(tag: 7466, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:42:17,209-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41185] for addition
-2013-11-06 17:42:17,209-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41185] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41185]
-2013-11-06 17:42:17,209-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41185]
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f9e
-2013-11-06 17:42:17,245-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f9e
-MetaChannel [context: cps-41185, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41185]
-2013-11-06 17:42:17,245-0600 INFO MetaChannel MetaChannel [context: cps-41185, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41185]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:42:17,245-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:42:17,286-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 17:42:17,366-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 17:42:17,408-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-crush.mcs.anl.gov:0 (quality: 18.0) submitting urn:job-1-1383759274741-1383759274742: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:42:17,408-0600 INFO Cpu crush.mcs.anl.gov:0 (quality: 18.0) submitting urn:job-1-1383759274741-1383759274742: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:42:17,416-0600 DEBUG Cpu Active workerid=crush.mcs.anl.gov:000000
-Handler(tag: 67, JOBSTATUS) unregistering (send)
-2013-11-06 17:42:17,416-0600 INFO RequestHandler Handler(tag: 67, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:42:17,423-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41186] for addition
-2013-11-06 17:42:17,424-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41186] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41186]
-2013-11-06 17:42:17,424-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41186]
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f9d
-2013-11-06 17:42:17,447-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f9d
-MetaChannel [context: cps-41186, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41186]
-2013-11-06 17:42:17,447-0600 INFO MetaChannel MetaChannel [context: cps-41186, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41186]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:42:17,447-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:42:17,487-0600 INFO RequestHandler Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-06 17:42:17,568-0600 INFO RequestHandler Handler(tag: 42566, SUBMITJOB) unregistering (send)
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:42:17,996-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:42:19,174-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:42:19,174-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:42:19,174-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:42:19,174-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:42:19,175-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41184] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:42:19,175-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41184] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 7467, JOBSTATUS) unregistering (send)
-2013-11-06 17:42:19,175-0600 INFO RequestHandler Handler(tag: 7467, JOBSTATUS) unregistering (send)
-2013-11-06 17:42:19,175-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-null: Channel shut down
-2013-11-06 17:42:19,175-0600 INFO AbstractTCPChannel null: Channel shut down
-grind.mcs.anl.gov:0 (quality: 68.0) submitting urn:job-2-1383759274744-1383759274745: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:42:19,175-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 68.0) submitting urn:job-2-1383759274744-1383759274745: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-MetaChannel [context: cps-41184, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41184]] binding to NullChannel
-2013-11-06 17:42:19,175-0600 INFO MetaChannel MetaChannel [context: cps-41184, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41184]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41184]
-2013-11-06 17:42:19,175-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41184]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:42:19,176-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:42:19,176-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:42:19,176-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:42:19,176-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:42:19,176-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:42:19,176-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:42:19,176-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:42:19,176-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:42:19,176-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:42:19,176-0600 INFO ChannelManager Channel exception handled
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:42:19,177-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:42:19,177-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41186] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:42:19,177-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41186] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-06 17:42:19,177-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-41186, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41186]] binding to NullChannel
-2013-11-06 17:42:19,177-0600 INFO MetaChannel MetaChannel [context: cps-41186, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41186]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41186]
-2013-11-06 17:42:19,177-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41186]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:42:19,177-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:42:19,177-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:42:19,177-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:42:19,177-0600 INFO ChannelManager Channel exception handled
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:42:19,180-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:42:19,181-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:42:19,181-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7468, JOBSTATUS) unregistering (send)
-2013-11-06 17:42:19,181-0600 INFO RequestHandler Handler(tag: 7468, JOBSTATUS) unregistering (send)
-Channel is closed. Removing.
-2013-11-06 17:42:19,193-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:42:19,193-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:42:19,193-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41186] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:42:19,193-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41186] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:42:19,193-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41186]
-2013-11-06 17:42:19,193-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41186]
-Channel is closed. Removing.
-2013-11-06 17:42:19,193-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:42:19,193-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:42:19,193-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41184] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:42:19,193-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41184] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:42:19,194-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41184]
-2013-11-06 17:42:19,194-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41184]
-2013-11-06 17:42:19,427-0600 DEBUG Cpu Completed
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:42:19,427-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-crush.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:42:19,427-0600 INFO Cpu crush.mcs.anl.gov:0 jobTerminated
-Channel config: null -> {keepalive=null}
-2013-11-06 17:42:19,427-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-06 17:42:19,427-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-TCPChannel [type: server, contact: 127.0.0.1:41185] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:42:19,427-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41185] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:42:19,427-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-Handler(tag: 68, JOBSTATUS) unregistering (send)
-2013-11-06 17:42:19,427-0600 INFO RequestHandler Handler(tag: 68, JOBSTATUS) unregistering (send)
-2013-11-06 17:42:19,427-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-null: Channel shut down
-2013-11-06 17:42:19,428-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-41185, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41185]] binding to NullChannel
-2013-11-06 17:42:19,428-0600 INFO MetaChannel MetaChannel [context: cps-41185, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41185]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41185]
-2013-11-06 17:42:19,428-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41185]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:42:19,428-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:42:19,428-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:42:19,428-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:42:19,428-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:42:19,428-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:42:19,428-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:42:19,428-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:42:19,429-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:42:19,429-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:42:19,429-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 17:42:19,434-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:42:19,434-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:42:19,435-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41185] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:42:19,435-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41185] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:42:19,435-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41185]
-2013-11-06 17:42:19,435-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41185]
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:42:21,192-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:42:21,192-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:42:21,192-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:42:21,192-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-06 17:42:21,192-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7469, JOBSTATUS) unregistering (send)
-2013-11-06 17:42:21,192-0600 INFO RequestHandler Handler(tag: 7469, JOBSTATUS) unregistering (send)
-2013-11-06 17:42:21,192-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-No streams
-2013-11-06 17:42:23,446-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-06 17:42:25,024-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-No streams
-2013-11-06 17:42:33,454-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:42:43,462-0600 INFO Multiplexer No streams
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:42:43,628-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41187] for addition
-2013-11-06 17:42:43,628-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41187] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41187]
-2013-11-06 17:42:43,628-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41187]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f9c
-2013-11-06 17:42:43,663-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f9c
-MetaChannel [context: cps-41187, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41187]
-2013-11-06 17:42:43,663-0600 INFO MetaChannel MetaChannel [context: cps-41187, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41187]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:42:43,663-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:42:43,704-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 17:42:43,704-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 17:42:43,784-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 17:42:43,818-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 69.0) submitting urn:job-0-1383759274747-1383759274748: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:42:43,818-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 69.0) submitting urn:job-0-1383759274747-1383759274748: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:42:43,819-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 17:42:43,819-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-06 17:42:43,825-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7470, JOBSTATUS) unregistering (send)
-2013-11-06 17:42:43,825-0600 INFO RequestHandler Handler(tag: 7470, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:42:43,844-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41188] for addition
-2013-11-06 17:42:43,844-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41188] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41188]
-2013-11-06 17:42:43,844-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41188]
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f9b
-2013-11-06 17:42:43,865-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f9b
-MetaChannel [context: cps-41188, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41188]
-2013-11-06 17:42:43,865-0600 INFO MetaChannel MetaChannel [context: cps-41188, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41188]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:42:43,865-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:42:43,906-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 17:42:43,986-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 17:42:44,019-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-crush.mcs.anl.gov:0 (quality: 19.0) submitting urn:job-1-1383759274750-1383759274751: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:42:44,019-0600 INFO Cpu crush.mcs.anl.gov:0 (quality: 19.0) submitting urn:job-1-1383759274750-1383759274751: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:42:44,026-0600 DEBUG Cpu Active workerid=crush.mcs.anl.gov:000000
-Handler(tag: 69, JOBSTATUS) unregistering (send)
-2013-11-06 17:42:44,026-0600 INFO RequestHandler Handler(tag: 69, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:42:44,043-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41189] for addition
-2013-11-06 17:42:44,044-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41189] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41189]
-2013-11-06 17:42:44,044-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41189]
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f9a
-2013-11-06 17:42:44,067-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f9a
-MetaChannel [context: cps-41189, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41189]
-2013-11-06 17:42:44,067-0600 INFO MetaChannel MetaChannel [context: cps-41189, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41189]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:42:44,067-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:42:44,107-0600 INFO RequestHandler Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-06 17:42:44,188-0600 INFO RequestHandler Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-06 17:42:45,897-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:42:45,897-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:42:45,897-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:42:45,897-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Channel config: null -> {keepalive=null}
-2013-11-06 17:42:45,897-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-06 17:42:45,897-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-Handler(tag: 7471, JOBSTATUS) unregistering (send)
-2013-11-06 17:42:45,898-0600 INFO RequestHandler Handler(tag: 7471, JOBSTATUS) unregistering (send)
-TCPChannel [type: server, contact: 127.0.0.1:41187] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:42:45,897-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41187] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-grind.mcs.anl.gov:0 (quality: 70.0) submitting urn:job-2-1383759274753-1383759274754: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:42:45,898-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 70.0) submitting urn:job-2-1383759274753-1383759274754: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-null: Channel shut down
-2013-11-06 17:42:45,898-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-41187, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41187]] binding to NullChannel
-2013-11-06 17:42:45,898-0600 INFO MetaChannel MetaChannel [context: cps-41187, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41187]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41187]
-2013-11-06 17:42:45,899-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41187]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:42:45,899-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:42:45,899-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:42:45,899-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:42:45,899-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:42:45,899-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:42:45,899-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:42:45,899-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:42:45,899-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:42:45,899-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:42:45,899-0600 INFO ChannelManager Channel exception handled
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:42:45,900-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:42:45,900-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41189] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:42:45,900-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41189] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-06 17:42:45,900-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-41189, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41189]] binding to NullChannel
-2013-11-06 17:42:45,900-0600 INFO MetaChannel MetaChannel [context: cps-41189, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41189]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41189]
-2013-11-06 17:42:45,900-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41189]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:42:45,900-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:42:45,900-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:42:45,900-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:42:45,900-0600 INFO ChannelManager Channel exception handled
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:42:45,902-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:42:45,904-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:42:45,904-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7472, JOBSTATUS) unregistering (send)
-2013-11-06 17:42:45,904-0600 INFO RequestHandler Handler(tag: 7472, JOBSTATUS) unregistering (send)
-Channel is closed. Removing.
-2013-11-06 17:42:45,913-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:42:45,913-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:42:45,913-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41187] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:42:45,913-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41187] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:42:45,913-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41187]
-2013-11-06 17:42:45,913-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41187]
-Channel is closed. Removing.
-2013-11-06 17:42:45,913-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:42:45,913-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:42:45,913-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41189] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:42:45,913-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41189] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:42:45,913-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41189]
-2013-11-06 17:42:45,914-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41189]
-2013-11-06 17:42:46,042-0600 DEBUG Cpu Completed
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:42:46,043-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-crush.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:42:46,043-0600 INFO Cpu crush.mcs.anl.gov:0 jobTerminated
-Channel config: null -> {keepalive=null}
-2013-11-06 17:42:46,043-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-06 17:42:46,043-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-TCPChannel [type: server, contact: 127.0.0.1:41188] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:42:46,043-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41188] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 70, JOBSTATUS) unregistering (send)
-2013-11-06 17:42:46,043-0600 INFO RequestHandler Handler(tag: 70, JOBSTATUS) unregistering (send)
-2013-11-06 17:42:46,043-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-null: Channel shut down
-2013-11-06 17:42:46,046-0600 INFO AbstractTCPChannel null: Channel shut down
-2013-11-06 17:42:46,047-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-MetaChannel [context: cps-41188, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41188]] binding to NullChannel
-2013-11-06 17:42:46,047-0600 INFO MetaChannel MetaChannel [context: cps-41188, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41188]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41188]
-2013-11-06 17:42:46,047-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41188]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:42:46,047-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:42:46,047-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:42:46,047-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:42:46,047-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:42:46,047-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:42:46,047-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:42:46,047-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:42:46,047-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:42:46,048-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:42:46,048-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 17:42:46,054-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:42:46,054-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:42:46,054-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41188] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:42:46,054-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41188] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:42:46,054-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41188]
-2013-11-06 17:42:46,054-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41188]
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:42:47,912-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:42:47,912-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:42:47,912-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:42:47,912-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-06 17:42:47,912-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7473, JOBSTATUS) unregistering (send)
-2013-11-06 17:42:47,912-0600 INFO RequestHandler Handler(tag: 7473, JOBSTATUS) unregistering (send)
-2013-11-06 17:42:47,913-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-No streams
-2013-11-06 17:42:53,475-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:42:54,470-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174059.736, now=131106-174300.585, channel=TCPChannel [type: server, contact: 127.0.0.1:41177]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:43:00,585-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174059.736, now=131106-174300.585, channel=TCPChannel [type: server, contact: 127.0.0.1:41177]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:43:00,585-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174059.736, now=131106-174300.585, channel=TCPChannel [type: server, contact: 127.0.0.1:41177]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:43:00,585-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174059.736, now=131106-174300.585, channel=TCPChannel [type: server, contact: 127.0.0.1:41177]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:43:00,585-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174059.736, now=131106-174300.585, channel=TCPChannel [type: server, contact: 127.0.0.1:41177]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:43:00,585-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174059.736, now=131106-174300.585, channel=TCPChannel [type: server, contact: 127.0.0.1:41177]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:43:00,586-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174059.736, now=131106-174300.585, channel=TCPChannel [type: server, contact: 127.0.0.1:41177]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:43:00,586-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174059.736, now=131106-174300.585, channel=TCPChannel [type: server, contact: 127.0.0.1:41177]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:43:00,586-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174059.736, now=131106-174300.585, channel=TCPChannel [type: server, contact: 127.0.0.1:41177]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:43:00,586-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174059.736, now=131106-174300.585, channel=TCPChannel [type: server, contact: 127.0.0.1:41177]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:43:00,586-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41177] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174059.736, now=131106-174300.585, channel=TCPChannel [type: server, contact: 127.0.0.1:41177]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:43:00,586-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41177] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174059.736, now=131106-174300.585, channel=TCPChannel [type: server, contact: 127.0.0.1:41177]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:43:00,586-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174059.938, now=131106-174300.796, channel=TCPChannel [type: server, contact: 127.0.0.1:41178]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:43:00,796-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174059.938, now=131106-174300.796, channel=TCPChannel [type: server, contact: 127.0.0.1:41178]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:43:00,796-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174059.938, now=131106-174300.796, channel=TCPChannel [type: server, contact: 127.0.0.1:41178]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:43:00,796-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174059.938, now=131106-174300.796, channel=TCPChannel [type: server, contact: 127.0.0.1:41178]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:43:00,796-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174059.938, now=131106-174300.796, channel=TCPChannel [type: server, contact: 127.0.0.1:41178]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:43:00,797-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174059.938, now=131106-174300.796, channel=TCPChannel [type: server, contact: 127.0.0.1:41178]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:43:00,797-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174059.938, now=131106-174300.796, channel=TCPChannel [type: server, contact: 127.0.0.1:41178]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:43:00,797-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174059.938, now=131106-174300.796, channel=TCPChannel [type: server, contact: 127.0.0.1:41178]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:43:00,797-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174059.938, now=131106-174300.796, channel=TCPChannel [type: server, contact: 127.0.0.1:41178]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:43:00,797-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174059.938, now=131106-174300.796, channel=TCPChannel [type: server, contact: 127.0.0.1:41178]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:43:00,797-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41178] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174059.938, now=131106-174300.796, channel=TCPChannel [type: server, contact: 127.0.0.1:41178]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:43:00,797-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41178] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174059.938, now=131106-174300.796, channel=TCPChannel [type: server, contact: 127.0.0.1:41178]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:43:00,797-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174100.140, now=131106-174300.996, channel=TCPChannel [type: server, contact: 127.0.0.1:41179]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:43:00,996-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174100.140, now=131106-174300.996, channel=TCPChannel [type: server, contact: 127.0.0.1:41179]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:43:00,996-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174100.140, now=131106-174300.996, channel=TCPChannel [type: server, contact: 127.0.0.1:41179]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:43:00,996-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174100.140, now=131106-174300.996, channel=TCPChannel [type: server, contact: 127.0.0.1:41179]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:43:00,997-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41179] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174100.140, now=131106-174300.996, channel=TCPChannel [type: server, contact: 127.0.0.1:41179]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:43:00,997-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41179] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174100.140, now=131106-174300.996, channel=TCPChannel [type: server, contact: 127.0.0.1:41179]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:43:00,997-0600 INFO ChannelManager Channel already shut down
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-06 17:43:02,341-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-No streams
-2013-11-06 17:43:03,482-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:43:13,490-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:43:23,497-0600 INFO Multiplexer No streams
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174126.878, now=131106-174327.724, channel=TCPChannel [type: server, contact: 127.0.0.1:41180]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:43:27,724-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174126.878, now=131106-174327.724, channel=TCPChannel [type: server, contact: 127.0.0.1:41180]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:43:27,724-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174126.878, now=131106-174327.724, channel=TCPChannel [type: server, contact: 127.0.0.1:41180]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:43:27,724-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174126.878, now=131106-174327.724, channel=TCPChannel [type: server, contact: 127.0.0.1:41180]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:43:27,724-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174126.878, now=131106-174327.724, channel=TCPChannel [type: server, contact: 127.0.0.1:41180]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:43:27,724-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174126.878, now=131106-174327.724, channel=TCPChannel [type: server, contact: 127.0.0.1:41180]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:43:27,724-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174126.878, now=131106-174327.724, channel=TCPChannel [type: server, contact: 127.0.0.1:41180]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:43:27,724-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174126.878, now=131106-174327.724, channel=TCPChannel [type: server, contact: 127.0.0.1:41180]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:43:27,724-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174126.878, now=131106-174327.724, channel=TCPChannel [type: server, contact: 127.0.0.1:41180]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:43:27,724-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174126.878, now=131106-174327.724, channel=TCPChannel [type: server, contact: 127.0.0.1:41180]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:43:27,724-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41180] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174126.878, now=131106-174327.724, channel=TCPChannel [type: server, contact: 127.0.0.1:41180]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:43:27,724-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41180] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174126.878, now=131106-174327.724, channel=TCPChannel [type: server, contact: 127.0.0.1:41180]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:43:27,724-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174127.079, now=131106-174327.937, channel=TCPChannel [type: server, contact: 127.0.0.1:41181]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:43:27,937-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174127.079, now=131106-174327.937, channel=TCPChannel [type: server, contact: 127.0.0.1:41181]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:43:27,937-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174127.079, now=131106-174327.937, channel=TCPChannel [type: server, contact: 127.0.0.1:41181]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:43:27,937-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174127.079, now=131106-174327.937, channel=TCPChannel [type: server, contact: 127.0.0.1:41181]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:43:27,937-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174127.079, now=131106-174327.937, channel=TCPChannel [type: server, contact: 127.0.0.1:41181]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:43:27,937-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174127.079, now=131106-174327.937, channel=TCPChannel [type: server, contact: 127.0.0.1:41181]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:43:27,937-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174127.079, now=131106-174327.937, channel=TCPChannel [type: server, contact: 127.0.0.1:41181]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:43:27,937-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174127.079, now=131106-174327.937, channel=TCPChannel [type: server, contact: 127.0.0.1:41181]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:43:27,937-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174127.079, now=131106-174327.937, channel=TCPChannel [type: server, contact: 127.0.0.1:41181]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:43:27,937-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174127.079, now=131106-174327.937, channel=TCPChannel [type: server, contact: 127.0.0.1:41181]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:43:27,937-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41181] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174127.079, now=131106-174327.937, channel=TCPChannel [type: server, contact: 127.0.0.1:41181]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:43:27,937-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41181] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174127.079, now=131106-174327.937, channel=TCPChannel [type: server, contact: 127.0.0.1:41181]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:43:27,937-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174127.281, now=131106-174328.141, channel=TCPChannel [type: server, contact: 127.0.0.1:41182]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:43:28,141-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174127.281, now=131106-174328.141, channel=TCPChannel [type: server, contact: 127.0.0.1:41182]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:43:28,141-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174127.281, now=131106-174328.141, channel=TCPChannel [type: server, contact: 127.0.0.1:41182]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:43:28,141-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174127.281, now=131106-174328.141, channel=TCPChannel [type: server, contact: 127.0.0.1:41182]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:43:28,141-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41182] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174127.281, now=131106-174328.141, channel=TCPChannel [type: server, contact: 127.0.0.1:41182]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:43:28,141-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41182] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174127.281, now=131106-174328.141, channel=TCPChannel [type: server, contact: 127.0.0.1:41182]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:43:28,141-0600 INFO ChannelManager Channel already shut down
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 17:43:30,944-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 17:43:33,505-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:43:39,660-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:43:43,513-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:43:53,521-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:44:03,529-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 17:44:07,418-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 17:44:13,537-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-06 17:44:16,978-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174217.144, now=131106-174417.988, channel=TCPChannel [type: server, contact: 127.0.0.1:41184]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:44:17,988-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174217.144, now=131106-174417.988, channel=TCPChannel [type: server, contact: 127.0.0.1:41184]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:44:17,989-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174217.144, now=131106-174417.988, channel=TCPChannel [type: server, contact: 127.0.0.1:41184]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:44:17,989-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174217.144, now=131106-174417.988, channel=TCPChannel [type: server, contact: 127.0.0.1:41184]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:44:17,989-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174217.144, now=131106-174417.988, channel=TCPChannel [type: server, contact: 127.0.0.1:41184]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:44:17,989-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174217.144, now=131106-174417.988, channel=TCPChannel [type: server, contact: 127.0.0.1:41184]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:44:17,989-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174217.144, now=131106-174417.988, channel=TCPChannel [type: server, contact: 127.0.0.1:41184]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:44:17,989-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174217.144, now=131106-174417.988, channel=TCPChannel [type: server, contact: 127.0.0.1:41184]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:44:17,989-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174217.144, now=131106-174417.988, channel=TCPChannel [type: server, contact: 127.0.0.1:41184]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:44:17,989-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174217.144, now=131106-174417.988, channel=TCPChannel [type: server, contact: 127.0.0.1:41184]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:44:17,989-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41184] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174217.144, now=131106-174417.988, channel=TCPChannel [type: server, contact: 127.0.0.1:41184]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:44:17,989-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41184] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174217.144, now=131106-174417.988, channel=TCPChannel [type: server, contact: 127.0.0.1:41184]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:44:17,989-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174217.366, now=131106-174418.209, channel=TCPChannel [type: server, contact: 127.0.0.1:41185]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:44:18,210-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174217.366, now=131106-174418.209, channel=TCPChannel [type: server, contact: 127.0.0.1:41185]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:44:18,210-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174217.366, now=131106-174418.209, channel=TCPChannel [type: server, contact: 127.0.0.1:41185]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:44:18,210-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174217.366, now=131106-174418.209, channel=TCPChannel [type: server, contact: 127.0.0.1:41185]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:44:18,210-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174217.366, now=131106-174418.209, channel=TCPChannel [type: server, contact: 127.0.0.1:41185]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:44:18,210-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174217.366, now=131106-174418.209, channel=TCPChannel [type: server, contact: 127.0.0.1:41185]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:44:18,210-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174217.366, now=131106-174418.209, channel=TCPChannel [type: server, contact: 127.0.0.1:41185]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:44:18,210-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174217.366, now=131106-174418.209, channel=TCPChannel [type: server, contact: 127.0.0.1:41185]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:44:18,210-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174217.366, now=131106-174418.209, channel=TCPChannel [type: server, contact: 127.0.0.1:41185]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:44:18,210-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174217.366, now=131106-174418.209, channel=TCPChannel [type: server, contact: 127.0.0.1:41185]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:44:18,210-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41185] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174217.366, now=131106-174418.209, channel=TCPChannel [type: server, contact: 127.0.0.1:41185]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:44:18,210-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41185] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174217.366, now=131106-174418.209, channel=TCPChannel [type: server, contact: 127.0.0.1:41185]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:44:18,210-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174217.568, now=131106-174418.424, channel=TCPChannel [type: server, contact: 127.0.0.1:41186]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:44:18,424-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174217.568, now=131106-174418.424, channel=TCPChannel [type: server, contact: 127.0.0.1:41186]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:44:18,425-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174217.568, now=131106-174418.424, channel=TCPChannel [type: server, contact: 127.0.0.1:41186]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:44:18,425-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174217.568, now=131106-174418.424, channel=TCPChannel [type: server, contact: 127.0.0.1:41186]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:44:18,425-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41186] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174217.568, now=131106-174418.424, channel=TCPChannel [type: server, contact: 127.0.0.1:41186]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:44:18,425-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41186] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174217.568, now=131106-174418.424, channel=TCPChannel [type: server, contact: 127.0.0.1:41186]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:44:18,425-0600 INFO ChannelManager Channel already shut down
-No streams
-2013-11-06 17:44:23,544-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:44:33,552-0600 INFO Multiplexer No streams
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:44:40,430-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41190] for addition
-2013-11-06 17:44:40,430-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41190] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41190]
-2013-11-06 17:44:40,430-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41190]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f99
-2013-11-06 17:44:40,452-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f99
-MetaChannel [context: cps-41190, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41190]
-2013-11-06 17:44:40,452-0600 INFO MetaChannel MetaChannel [context: cps-41190, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41190]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:44:40,452-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 17:44:40,493-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:44:40,493-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 17:44:40,573-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 17:44:40,589-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 71.0) submitting urn:job-0-1383759274756-1383759274757: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:44:40,589-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 71.0) submitting urn:job-0-1383759274756-1383759274757: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:44:40,589-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 17:44:40,589-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-06 17:44:40,596-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7474, JOBSTATUS) unregistering (send)
-2013-11-06 17:44:40,596-0600 INFO RequestHandler Handler(tag: 7474, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:44:40,632-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41191] for addition
-2013-11-06 17:44:40,632-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41191] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41191]
-2013-11-06 17:44:40,632-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41191]
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f98
-2013-11-06 17:44:40,655-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f98
-MetaChannel [context: cps-41191, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41191]
-2013-11-06 17:44:40,655-0600 INFO MetaChannel MetaChannel [context: cps-41191, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41191]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:44:40,655-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:44:40,696-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 17:44:40,776-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 17:44:40,790-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-crush.mcs.anl.gov:0 (quality: 20.0) submitting urn:job-1-1383759274759-1383759274760: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:44:40,790-0600 INFO Cpu crush.mcs.anl.gov:0 (quality: 20.0) submitting urn:job-1-1383759274759-1383759274760: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:44:40,796-0600 DEBUG Cpu Active workerid=crush.mcs.anl.gov:000000
-Handler(tag: 71, JOBSTATUS) unregistering (send)
-2013-11-06 17:44:40,796-0600 INFO RequestHandler Handler(tag: 71, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:44:40,841-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41192] for addition
-2013-11-06 17:44:40,841-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41192] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41192]
-2013-11-06 17:44:40,841-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41192]
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f97
-2013-11-06 17:44:40,877-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f97
-MetaChannel [context: cps-41192, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41192]
-2013-11-06 17:44:40,877-0600 INFO MetaChannel MetaChannel [context: cps-41192, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41192]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:44:40,877-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:44:40,918-0600 INFO RequestHandler Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-06 17:44:40,998-0600 INFO RequestHandler Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-06 17:44:42,650-0600 DEBUG Cpu Completed
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:44:42,650-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:44:42,650-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-Channel config: null -> {keepalive=null}
-2013-11-06 17:44:42,651-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-06 17:44:42,651-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-TCPChannel [type: server, contact: 127.0.0.1:41190] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:44:42,651-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41190] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 7475, JOBSTATUS) unregistering (send)
-2013-11-06 17:44:42,651-0600 INFO RequestHandler Handler(tag: 7475, JOBSTATUS) unregistering (send)
-2013-11-06 17:44:42,651-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 72.0) submitting urn:job-2-1383759274762-1383759274763: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:44:42,651-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 72.0) submitting urn:job-2-1383759274762-1383759274763: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-null: Channel shut down
-2013-11-06 17:44:42,651-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-41190, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41190]] binding to NullChannel
-2013-11-06 17:44:42,652-0600 INFO MetaChannel MetaChannel [context: cps-41190, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41190]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41190]
-2013-11-06 17:44:42,652-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41190]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:44:42,652-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:44:42,652-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:44:42,652-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:44:42,652-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:44:42,652-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:44:42,652-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:44:42,652-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:44:42,653-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:44:42,653-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:44:42,653-0600 INFO ChannelManager Channel exception handled
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:44:42,653-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:44:42,653-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41192] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:44:42,653-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41192] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-06 17:44:42,653-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-41192, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41192]] binding to NullChannel
-2013-11-06 17:44:42,653-0600 INFO MetaChannel MetaChannel [context: cps-41192, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41192]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41192]
-2013-11-06 17:44:42,654-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41192]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:44:42,654-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:44:42,654-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:44:42,654-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:44:42,654-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 17:44:42,663-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:44:42,663-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:44:42,663-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41190] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:44:42,663-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41190] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:44:42,663-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41190]
-2013-11-06 17:44:42,663-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41190]
-Channel is closed. Removing.
-2013-11-06 17:44:42,663-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:44:42,664-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:44:42,664-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41192] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:44:42,664-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41192] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:44:42,664-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41192]
-2013-11-06 17:44:42,664-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41192]
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:44:42,665-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:44:42,666-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:44:42,666-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7476, JOBSTATUS) unregistering (send)
-2013-11-06 17:44:42,666-0600 INFO RequestHandler Handler(tag: 7476, JOBSTATUS) unregistering (send)
-2013-11-06 17:44:42,827-0600 DEBUG Cpu Completed
-crush.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:44:42,827-0600 INFO Cpu crush.mcs.anl.gov:0 jobTerminated
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:44:42,827-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:44:42,827-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-Channel config: null -> {keepalive=null}
-2013-11-06 17:44:42,827-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-Handler(tag: 72, JOBSTATUS) unregistering (send)
-2013-11-06 17:44:42,827-0600 INFO RequestHandler Handler(tag: 72, JOBSTATUS) unregistering (send)
-2013-11-06 17:44:42,827-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 17:44:42,827-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-TCPChannel [type: server, contact: 127.0.0.1:41191] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:44:42,827-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41191] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-06 17:44:42,828-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-41191, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41191]] binding to NullChannel
-2013-11-06 17:44:42,828-0600 INFO MetaChannel MetaChannel [context: cps-41191, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41191]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41191]
-2013-11-06 17:44:42,828-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41191]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:44:42,828-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:44:42,828-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:44:42,828-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:44:42,828-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:44:42,828-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:44:42,828-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:44:42,828-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:44:42,828-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:44:42,828-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:44:42,829-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 17:44:42,844-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:44:42,844-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:44:42,845-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41191] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:44:42,845-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41191] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:44:42,845-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41191]
-2013-11-06 17:44:42,845-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41191]
-No streams
-2013-11-06 17:44:43,567-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:44:43,892-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174243.784, now=131106-174444.628, channel=TCPChannel [type: server, contact: 127.0.0.1:41187]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:44:44,628-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174243.784, now=131106-174444.628, channel=TCPChannel [type: server, contact: 127.0.0.1:41187]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:44:44,629-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174243.784, now=131106-174444.628, channel=TCPChannel [type: server, contact: 127.0.0.1:41187]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:44:44,629-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174243.784, now=131106-174444.628, channel=TCPChannel [type: server, contact: 127.0.0.1:41187]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:44:44,629-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174243.784, now=131106-174444.628, channel=TCPChannel [type: server, contact: 127.0.0.1:41187]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:44:44,629-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174243.784, now=131106-174444.628, channel=TCPChannel [type: server, contact: 127.0.0.1:41187]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:44:44,629-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174243.784, now=131106-174444.628, channel=TCPChannel [type: server, contact: 127.0.0.1:41187]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:44:44,629-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174243.784, now=131106-174444.628, channel=TCPChannel [type: server, contact: 127.0.0.1:41187]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:44:44,629-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174243.784, now=131106-174444.628, channel=TCPChannel [type: server, contact: 127.0.0.1:41187]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:44:44,629-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174243.784, now=131106-174444.628, channel=TCPChannel [type: server, contact: 127.0.0.1:41187]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:44:44,629-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41187] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174243.784, now=131106-174444.628, channel=TCPChannel [type: server, contact: 127.0.0.1:41187]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:44:44,629-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41187] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174243.784, now=131106-174444.628, channel=TCPChannel [type: server, contact: 127.0.0.1:41187]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:44:44,629-0600 INFO ChannelManager Channel already shut down
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:44:44,678-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:44:44,678-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:44:44,678-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:44:44,678-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-06 17:44:44,678-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7477, JOBSTATUS) unregistering (send)
-2013-11-06 17:44:44,678-0600 INFO RequestHandler Handler(tag: 7477, JOBSTATUS) unregistering (send)
-2013-11-06 17:44:44,678-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174243.986, now=131106-174444.844, channel=TCPChannel [type: server, contact: 127.0.0.1:41188]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:44:44,844-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174243.986, now=131106-174444.844, channel=TCPChannel [type: server, contact: 127.0.0.1:41188]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:44:44,845-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174243.986, now=131106-174444.844, channel=TCPChannel [type: server, contact: 127.0.0.1:41188]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:44:44,845-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174243.986, now=131106-174444.844, channel=TCPChannel [type: server, contact: 127.0.0.1:41188]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:44:44,845-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174243.986, now=131106-174444.844, channel=TCPChannel [type: server, contact: 127.0.0.1:41188]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:44:44,845-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174243.986, now=131106-174444.844, channel=TCPChannel [type: server, contact: 127.0.0.1:41188]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:44:44,845-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174243.986, now=131106-174444.844, channel=TCPChannel [type: server, contact: 127.0.0.1:41188]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:44:44,845-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174243.986, now=131106-174444.844, channel=TCPChannel [type: server, contact: 127.0.0.1:41188]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:44:44,845-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174243.986, now=131106-174444.844, channel=TCPChannel [type: server, contact: 127.0.0.1:41188]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:44:44,845-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174243.986, now=131106-174444.844, channel=TCPChannel [type: server, contact: 127.0.0.1:41188]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:44:44,845-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41188] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174243.986, now=131106-174444.844, channel=TCPChannel [type: server, contact: 127.0.0.1:41188]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:44:44,845-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41188] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174243.986, now=131106-174444.844, channel=TCPChannel [type: server, contact: 127.0.0.1:41188]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:44:44,845-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174244.188, now=131106-174445.044, channel=TCPChannel [type: server, contact: 127.0.0.1:41189]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:44:45,044-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174244.188, now=131106-174445.044, channel=TCPChannel [type: server, contact: 127.0.0.1:41189]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:44:45,044-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174244.188, now=131106-174445.044, channel=TCPChannel [type: server, contact: 127.0.0.1:41189]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:44:45,044-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174244.188, now=131106-174445.044, channel=TCPChannel [type: server, contact: 127.0.0.1:41189]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:44:45,044-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41189] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174244.188, now=131106-174445.044, channel=TCPChannel [type: server, contact: 127.0.0.1:41189]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:44:45,045-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41189] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174244.188, now=131106-174445.044, channel=TCPChannel [type: server, contact: 127.0.0.1:41189]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:44:45,045-0600 INFO ChannelManager Channel already shut down
-No streams
-2013-11-06 17:44:53,575-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:44:54,296-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:45:00,173-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41193] for addition
-2013-11-06 17:45:00,173-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41193] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41193]
-2013-11-06 17:45:00,173-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41193]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f96
-2013-11-06 17:45:00,194-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f96
-MetaChannel [context: cps-41193, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41193]
-2013-11-06 17:45:00,194-0600 INFO MetaChannel MetaChannel [context: cps-41193, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41193]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:45:00,194-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 17:45:00,235-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:45:00,235-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 17:45:00,315-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 17:45:00,346-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 73.0) submitting urn:job-0-1383759274765-1383759274766: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:45:00,346-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 73.0) submitting urn:job-0-1383759274765-1383759274766: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:45:00,346-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 17:45:00,347-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-06 17:45:00,353-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7478, JOBSTATUS) unregistering (send)
-2013-11-06 17:45:00,353-0600 INFO RequestHandler Handler(tag: 7478, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:45:00,376-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41194] for addition
-2013-11-06 17:45:00,376-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41194] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41194]
-2013-11-06 17:45:00,376-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41194]
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f95
-2013-11-06 17:45:00,416-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f95
-MetaChannel [context: cps-41194, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41194]
-2013-11-06 17:45:00,416-0600 INFO MetaChannel MetaChannel [context: cps-41194, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41194]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:45:00,416-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:45:00,436-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 17:45:00,517-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 17:45:00,547-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-crush.mcs.anl.gov:0 (quality: 21.0) submitting urn:job-1-1383759274768-1383759274769: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:45:00,547-0600 INFO Cpu crush.mcs.anl.gov:0 (quality: 21.0) submitting urn:job-1-1383759274768-1383759274769: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:45:00,554-0600 DEBUG Cpu Active workerid=crush.mcs.anl.gov:000000
-Handler(tag: 73, JOBSTATUS) unregistering (send)
-2013-11-06 17:45:00,554-0600 INFO RequestHandler Handler(tag: 73, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:45:00,576-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41195] for addition
-2013-11-06 17:45:00,576-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41195] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41195]
-2013-11-06 17:45:00,576-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41195]
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f94
-2013-11-06 17:45:00,597-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f94
-MetaChannel [context: cps-41195, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41195]
-2013-11-06 17:45:00,598-0600 INFO MetaChannel MetaChannel [context: cps-41195, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41195]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:45:00,598-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:45:00,658-0600 INFO RequestHandler Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-06 17:45:00,739-0600 INFO RequestHandler Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-06 17:45:02,362-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:45:02,362-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:45:02,363-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:45:02,363-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 7479, JOBSTATUS) unregistering (send)
-2013-11-06 17:45:02,363-0600 INFO RequestHandler Handler(tag: 7479, JOBSTATUS) unregistering (send)
-2013-11-06 17:45:02,363-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-Channel config: null -> {keepalive=null}
-2013-11-06 17:45:02,363-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41193] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:45:02,363-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41193] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-grind.mcs.anl.gov:0 (quality: 74.0) submitting urn:job-2-1383759274771-1383759274772: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:45:02,363-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 74.0) submitting urn:job-2-1383759274771-1383759274772: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-null: Channel shut down
-2013-11-06 17:45:02,364-0600 INFO AbstractTCPChannel null: Channel shut down
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:45:02,364-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-MetaChannel [context: cps-41193, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41193]] binding to NullChannel
-2013-11-06 17:45:02,364-0600 INFO MetaChannel MetaChannel [context: cps-41193, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41193]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41193]
-2013-11-06 17:45:02,364-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41193]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:45:02,364-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:45:02,364-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:45:02,365-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:45:02,365-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:45:02,365-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:45:02,365-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:45:02,365-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:45:02,365-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:45:02,365-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:45:02,365-0600 INFO ChannelManager Channel exception handled
-Channel config: null -> {keepalive=null}
-2013-11-06 17:45:02,365-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:45:02,365-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-TCPChannel [type: server, contact: 127.0.0.1:41193] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:45:02,365-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41193] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:45:02,365-0600 INFO ChannelManager Channel already shut down
-Channel config: null -> {keepalive=null}
-2013-11-06 17:45:02,365-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41193]
-2013-11-06 17:45:02,365-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41193]
-TCPChannel [type: server, contact: 127.0.0.1:41195] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:45:02,366-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41195] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-06 17:45:02,366-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-41195, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41195]] binding to NullChannel
-2013-11-06 17:45:02,366-0600 INFO MetaChannel MetaChannel [context: cps-41195, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41195]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41195]
-2013-11-06 17:45:02,366-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41195]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:45:02,366-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:45:02,366-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:45:02,366-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:45:02,366-0600 INFO ChannelManager Channel exception handled
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:45:02,369-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:45:02,370-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:45:02,370-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7480, JOBSTATUS) unregistering (send)
-2013-11-06 17:45:02,371-0600 INFO RequestHandler Handler(tag: 7480, JOBSTATUS) unregistering (send)
-Channel is closed. Removing.
-2013-11-06 17:45:02,386-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:45:02,386-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:45:02,386-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41195] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:45:02,386-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41195] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:45:02,386-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41195]
-2013-11-06 17:45:02,386-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41195]
-2013-11-06 17:45:02,564-0600 DEBUG Cpu Completed
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:45:02,564-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-crush.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:45:02,564-0600 INFO Cpu crush.mcs.anl.gov:0 jobTerminated
-Channel config: null -> {keepalive=null}
-2013-11-06 17:45:02,564-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-06 17:45:02,564-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-TCPChannel [type: server, contact: 127.0.0.1:41194] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:45:02,564-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41194] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 74, JOBSTATUS) unregistering (send)
-2013-11-06 17:45:02,564-0600 INFO RequestHandler Handler(tag: 74, JOBSTATUS) unregistering (send)
-2013-11-06 17:45:02,564-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-null: Channel shut down
-2013-11-06 17:45:02,564-0600 INFO AbstractTCPChannel null: Channel shut down
-2013-11-06 17:45:02,564-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-MetaChannel [context: cps-41194, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41194]] binding to NullChannel
-2013-11-06 17:45:02,564-0600 INFO MetaChannel MetaChannel [context: cps-41194, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41194]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41194]
-2013-11-06 17:45:02,564-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41194]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:45:02,564-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:45:02,565-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:45:02,565-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:45:02,565-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:45:02,565-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:45:02,565-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:45:02,565-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:45:02,565-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:45:02,565-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:45:02,565-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 17:45:02,567-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:45:02,567-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:45:02,567-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41194] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:45:02,567-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41194] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:45:02,567-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41194]
-2013-11-06 17:45:02,567-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41194]
-No streams
-2013-11-06 17:45:03,590-0600 INFO Multiplexer No streams
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:45:04,380-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:45:04,381-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:45:04,381-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:45:04,381-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-06 17:45:04,381-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7481, JOBSTATUS) unregistering (send)
-2013-11-06 17:45:04,381-0600 INFO RequestHandler Handler(tag: 7481, JOBSTATUS) unregistering (send)
-2013-11-06 17:45:04,381-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-No streams
-2013-11-06 17:45:13,598-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 17:45:20,367-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 17:45:23,606-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-06 17:45:31,614-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-06 17:45:33,614-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:45:43,622-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:45:53,630-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:45:56,840-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:46:03,638-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:46:08,932-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:46:13,646-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:46:23,654-0600 INFO Multiplexer No streams
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:46:29,641-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41196] for addition
-2013-11-06 17:46:29,641-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41196] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41196]
-2013-11-06 17:46:29,641-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41196]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f93
-2013-11-06 17:46:29,672-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f93
-MetaChannel [context: cps-41196, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41196]
-2013-11-06 17:46:29,672-0600 INFO MetaChannel MetaChannel [context: cps-41196, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41196]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:46:29,673-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:46:29,713-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 17:46:29,713-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 17:46:29,794-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 17:46:29,827-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 75.0) submitting urn:job-0-1383759274774-1383759274775: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:46:29,827-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 75.0) submitting urn:job-0-1383759274774-1383759274775: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:46:29,827-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 17:46:29,827-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-06 17:46:29,834-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7482, JOBSTATUS) unregistering (send)
-2013-11-06 17:46:29,834-0600 INFO RequestHandler Handler(tag: 7482, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:46:29,852-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41197] for addition
-2013-11-06 17:46:29,852-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41197] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41197]
-2013-11-06 17:46:29,852-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41197]
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f92
-2013-11-06 17:46:29,874-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f92
-MetaChannel [context: cps-41197, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41197]
-2013-11-06 17:46:29,874-0600 INFO MetaChannel MetaChannel [context: cps-41197, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41197]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:46:29,874-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:46:29,915-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 17:46:29,995-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 17:46:30,027-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-crush.mcs.anl.gov:0 (quality: 22.0) submitting urn:job-1-1383759274777-1383759274778: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:46:30,028-0600 INFO Cpu crush.mcs.anl.gov:0 (quality: 22.0) submitting urn:job-1-1383759274777-1383759274778: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:46:30,033-0600 DEBUG Cpu Active workerid=crush.mcs.anl.gov:000000
-Handler(tag: 75, JOBSTATUS) unregistering (send)
-2013-11-06 17:46:30,033-0600 INFO RequestHandler Handler(tag: 75, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:46:30,055-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41198] for addition
-2013-11-06 17:46:30,055-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41198] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41198]
-2013-11-06 17:46:30,055-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41198]
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f91
-2013-11-06 17:46:30,076-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f91
-MetaChannel [context: cps-41198, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41198]
-2013-11-06 17:46:30,076-0600 INFO MetaChannel MetaChannel [context: cps-41198, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41198]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:46:30,076-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:46:30,117-0600 INFO RequestHandler Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-06 17:46:30,197-0600 INFO RequestHandler Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-06 17:46:31,856-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:46:31,856-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:46:31,856-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:46:31,856-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Channel config: null -> {keepalive=null}
-2013-11-06 17:46:31,856-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-06 17:46:31,856-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-Handler(tag: 7483, JOBSTATUS) unregistering (send)
-2013-11-06 17:46:31,856-0600 INFO RequestHandler Handler(tag: 7483, JOBSTATUS) unregistering (send)
-TCPChannel [type: server, contact: 127.0.0.1:41196] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:46:31,856-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41196] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-grind.mcs.anl.gov:0 (quality: 76.0) submitting urn:job-2-1383759274780-1383759274781: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:46:31,856-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 76.0) submitting urn:job-2-1383759274780-1383759274781: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-null: Channel shut down
-2013-11-06 17:46:31,857-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-41196, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41196]] binding to NullChannel
-2013-11-06 17:46:31,857-0600 INFO MetaChannel MetaChannel [context: cps-41196, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41196]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41196]
-2013-11-06 17:46:31,857-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41196]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:46:31,857-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:46:31,857-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:46:31,857-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:46:31,857-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:46:31,857-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:46:31,858-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:46:31,858-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:46:31,858-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:46:31,858-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:46:31,858-0600 INFO ChannelManager Channel exception handled
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:46:31,858-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:46:31,858-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41198] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:46:31,858-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41198] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-06 17:46:31,858-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-41198, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41198]] binding to NullChannel
-2013-11-06 17:46:31,858-0600 INFO MetaChannel MetaChannel [context: cps-41198, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41198]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41198]
-2013-11-06 17:46:31,858-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41198]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:46:31,858-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:46:31,859-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:46:31,859-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:46:31,859-0600 INFO ChannelManager Channel exception handled
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:46:31,862-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-Channel is closed. Removing.
-2013-11-06 17:46:31,862-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:46:31,862-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:46:31,862-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:46:31,862-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Channel config: null -> {keepalive=null}
-2013-11-06 17:46:31,862-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-Handler(tag: 7484, JOBSTATUS) unregistering (send)
-2013-11-06 17:46:31,863-0600 INFO RequestHandler Handler(tag: 7484, JOBSTATUS) unregistering (send)
-TCPChannel [type: server, contact: 127.0.0.1:41198] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:46:31,863-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41198] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:46:31,863-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41198]
-2013-11-06 17:46:31,863-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41198]
-Channel is closed. Removing.
-2013-11-06 17:46:31,863-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:46:31,863-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:46:31,863-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41196] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:46:31,863-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41196] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:46:31,863-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41196]
-2013-11-06 17:46:31,863-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41196]
-2013-11-06 17:46:32,054-0600 DEBUG Cpu Completed
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:46:32,054-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-crush.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:46:32,054-0600 INFO Cpu crush.mcs.anl.gov:0 jobTerminated
-Channel config: null -> {keepalive=null}
-2013-11-06 17:46:32,054-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-06 17:46:32,054-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-TCPChannel [type: server, contact: 127.0.0.1:41197] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:46:32,054-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41197] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 76, JOBSTATUS) unregistering (send)
-2013-11-06 17:46:32,054-0600 INFO RequestHandler Handler(tag: 76, JOBSTATUS) unregistering (send)
-2013-11-06 17:46:32,054-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-null: Channel shut down
-2013-11-06 17:46:32,055-0600 INFO AbstractTCPChannel null: Channel shut down
-2013-11-06 17:46:32,055-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-MetaChannel [context: cps-41197, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41197]] binding to NullChannel
-2013-11-06 17:46:32,055-0600 INFO MetaChannel MetaChannel [context: cps-41197, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41197]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41197]
-2013-11-06 17:46:32,055-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41197]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:46:32,055-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:46:32,055-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:46:32,055-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:46:32,055-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:46:32,055-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:46:32,055-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:46:32,055-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:46:32,055-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:46:32,055-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:46:32,056-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 17:46:32,064-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:46:32,064-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:46:32,064-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41197] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:46:32,064-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41197] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:46:32,064-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41197]
-2013-11-06 17:46:32,064-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41197]
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:46:33,314-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:46:33,669-0600 INFO Multiplexer No streams
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:46:33,874-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:46:33,874-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:46:33,874-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:46:33,874-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-06 17:46:33,874-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-2013-11-06 17:46:33,874-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-Handler(tag: 7485, JOBSTATUS) unregistering (send)
-2013-11-06 17:46:33,874-0600 INFO RequestHandler Handler(tag: 7485, JOBSTATUS) unregistering (send)
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174440.573, now=131106-174641.430, channel=TCPChannel [type: server, contact: 127.0.0.1:41190]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:46:41,430-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174440.573, now=131106-174641.430, channel=TCPChannel [type: server, contact: 127.0.0.1:41190]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:46:41,430-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174440.573, now=131106-174641.430, channel=TCPChannel [type: server, contact: 127.0.0.1:41190]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:46:41,430-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174440.573, now=131106-174641.430, channel=TCPChannel [type: server, contact: 127.0.0.1:41190]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:46:41,430-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174440.573, now=131106-174641.430, channel=TCPChannel [type: server, contact: 127.0.0.1:41190]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:46:41,431-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174440.573, now=131106-174641.430, channel=TCPChannel [type: server, contact: 127.0.0.1:41190]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:46:41,431-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174440.573, now=131106-174641.430, channel=TCPChannel [type: server, contact: 127.0.0.1:41190]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:46:41,431-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174440.573, now=131106-174641.430, channel=TCPChannel [type: server, contact: 127.0.0.1:41190]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:46:41,431-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174440.573, now=131106-174641.430, channel=TCPChannel [type: server, contact: 127.0.0.1:41190]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:46:41,431-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174440.573, now=131106-174641.430, channel=TCPChannel [type: server, contact: 127.0.0.1:41190]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:46:41,431-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41190] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174440.573, now=131106-174641.430, channel=TCPChannel [type: server, contact: 127.0.0.1:41190]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:46:41,431-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41190] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174440.573, now=131106-174641.430, channel=TCPChannel [type: server, contact: 127.0.0.1:41190]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:46:41,431-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174440.776, now=131106-174641.632, channel=TCPChannel [type: server, contact: 127.0.0.1:41191]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:46:41,632-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174440.776, now=131106-174641.632, channel=TCPChannel [type: server, contact: 127.0.0.1:41191]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:46:41,632-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174440.776, now=131106-174641.632, channel=TCPChannel [type: server, contact: 127.0.0.1:41191]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:46:41,632-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174440.776, now=131106-174641.632, channel=TCPChannel [type: server, contact: 127.0.0.1:41191]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:46:41,632-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174440.776, now=131106-174641.632, channel=TCPChannel [type: server, contact: 127.0.0.1:41191]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:46:41,632-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174440.776, now=131106-174641.632, channel=TCPChannel [type: server, contact: 127.0.0.1:41191]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:46:41,632-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174440.776, now=131106-174641.632, channel=TCPChannel [type: server, contact: 127.0.0.1:41191]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:46:41,632-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174440.776, now=131106-174641.632, channel=TCPChannel [type: server, contact: 127.0.0.1:41191]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:46:41,633-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174440.776, now=131106-174641.632, channel=TCPChannel [type: server, contact: 127.0.0.1:41191]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:46:41,633-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174440.776, now=131106-174641.632, channel=TCPChannel [type: server, contact: 127.0.0.1:41191]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:46:41,633-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41191] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174440.776, now=131106-174641.632, channel=TCPChannel [type: server, contact: 127.0.0.1:41191]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:46:41,633-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41191] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174440.776, now=131106-174641.632, channel=TCPChannel [type: server, contact: 127.0.0.1:41191]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:46:41,633-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174440.998, now=131106-174641.841, channel=TCPChannel [type: server, contact: 127.0.0.1:41192]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:46:41,841-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174440.998, now=131106-174641.841, channel=TCPChannel [type: server, contact: 127.0.0.1:41192]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:46:41,841-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174440.998, now=131106-174641.841, channel=TCPChannel [type: server, contact: 127.0.0.1:41192]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:46:41,841-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174440.998, now=131106-174641.841, channel=TCPChannel [type: server, contact: 127.0.0.1:41192]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:46:41,841-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41192] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174440.998, now=131106-174641.841, channel=TCPChannel [type: server, contact: 127.0.0.1:41192]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:46:41,841-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41192] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174440.998, now=131106-174641.841, channel=TCPChannel [type: server, contact: 127.0.0.1:41192]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:46:41,841-0600 INFO ChannelManager Channel already shut down
-No streams
-2013-11-06 17:46:43,676-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:46:46,250-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:46:53,684-0600 INFO Multiplexer No streams
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:47:00,448-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41199] for addition
-2013-11-06 17:47:00,449-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41199] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41199]
-2013-11-06 17:47:00,449-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41199]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f90
-2013-11-06 17:47:00,484-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f90
-MetaChannel [context: cps-41199, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41199]
-2013-11-06 17:47:00,484-0600 INFO MetaChannel MetaChannel [context: cps-41199, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41199]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:47:00,484-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:47:00,524-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 17:47:00,524-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 17:47:00,605-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 17:47:00,654-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 77.0) submitting urn:job-0-1383759274783-1383759274784: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:47:00,655-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 77.0) submitting urn:job-0-1383759274783-1383759274784: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:47:00,655-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 17:47:00,655-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-06 17:47:00,661-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7486, JOBSTATUS) unregistering (send)
-2013-11-06 17:47:00,661-0600 INFO RequestHandler Handler(tag: 7486, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:47:00,664-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41200] for addition
-2013-11-06 17:47:00,664-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41200] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41200]
-2013-11-06 17:47:00,664-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41200]
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f8f
-2013-11-06 17:47:00,685-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f8f
-MetaChannel [context: cps-41200, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41200]
-2013-11-06 17:47:00,685-0600 INFO MetaChannel MetaChannel [context: cps-41200, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41200]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:47:00,685-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:47:00,726-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 17:47:00,806-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 17:47:00,856-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-crush.mcs.anl.gov:0 (quality: 23.0) submitting urn:job-1-1383759274786-1383759274787: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:47:00,856-0600 INFO Cpu crush.mcs.anl.gov:0 (quality: 23.0) submitting urn:job-1-1383759274786-1383759274787: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:47:00,861-0600 DEBUG Cpu Active workerid=crush.mcs.anl.gov:000000
-Handler(tag: 77, JOBSTATUS) unregistering (send)
-2013-11-06 17:47:00,861-0600 INFO RequestHandler Handler(tag: 77, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:47:00,864-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41201] for addition
-2013-11-06 17:47:00,864-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41201] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41201]
-2013-11-06 17:47:00,864-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41201]
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f8e
-2013-11-06 17:47:00,887-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f8e
-MetaChannel [context: cps-41201, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41201]
-2013-11-06 17:47:00,887-0600 INFO MetaChannel MetaChannel [context: cps-41201, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41201]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:47:00,887-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:47:00,931-0600 INFO RequestHandler Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-06 17:47:01,012-0600 INFO RequestHandler Handler(tag: 42566, SUBMITJOB) unregistering (send)
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174500.315, now=131106-174701.174, channel=TCPChannel [type: server, contact: 127.0.0.1:41193]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:47:01,174-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174500.315, now=131106-174701.174, channel=TCPChannel [type: server, contact: 127.0.0.1:41193]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:47:01,174-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174500.315, now=131106-174701.174, channel=TCPChannel [type: server, contact: 127.0.0.1:41193]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:47:01,174-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174500.315, now=131106-174701.174, channel=TCPChannel [type: server, contact: 127.0.0.1:41193]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:47:01,174-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174500.315, now=131106-174701.174, channel=TCPChannel [type: server, contact: 127.0.0.1:41193]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:47:01,174-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174500.315, now=131106-174701.174, channel=TCPChannel [type: server, contact: 127.0.0.1:41193]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:47:01,174-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174500.315, now=131106-174701.174, channel=TCPChannel [type: server, contact: 127.0.0.1:41193]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:47:01,174-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174500.315, now=131106-174701.174, channel=TCPChannel [type: server, contact: 127.0.0.1:41193]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:47:01,174-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174500.315, now=131106-174701.174, channel=TCPChannel [type: server, contact: 127.0.0.1:41193]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:47:01,174-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174500.315, now=131106-174701.174, channel=TCPChannel [type: server, contact: 127.0.0.1:41193]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:47:01,175-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41193] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174500.315, now=131106-174701.174, channel=TCPChannel [type: server, contact: 127.0.0.1:41193]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:47:01,175-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41193] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174500.315, now=131106-174701.174, channel=TCPChannel [type: server, contact: 127.0.0.1:41193]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:47:01,175-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174500.517, now=131106-174701.376, channel=TCPChannel [type: server, contact: 127.0.0.1:41194]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:47:01,376-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174500.517, now=131106-174701.376, channel=TCPChannel [type: server, contact: 127.0.0.1:41194]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:47:01,376-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174500.517, now=131106-174701.376, channel=TCPChannel [type: server, contact: 127.0.0.1:41194]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:47:01,376-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174500.517, now=131106-174701.376, channel=TCPChannel [type: server, contact: 127.0.0.1:41194]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:47:01,376-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174500.517, now=131106-174701.376, channel=TCPChannel [type: server, contact: 127.0.0.1:41194]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:47:01,376-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174500.517, now=131106-174701.376, channel=TCPChannel [type: server, contact: 127.0.0.1:41194]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:47:01,376-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174500.517, now=131106-174701.376, channel=TCPChannel [type: server, contact: 127.0.0.1:41194]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:47:01,376-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174500.517, now=131106-174701.376, channel=TCPChannel [type: server, contact: 127.0.0.1:41194]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:47:01,376-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174500.517, now=131106-174701.376, channel=TCPChannel [type: server, contact: 127.0.0.1:41194]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:47:01,376-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174500.517, now=131106-174701.376, channel=TCPChannel [type: server, contact: 127.0.0.1:41194]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:47:01,376-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41194] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174500.517, now=131106-174701.376, channel=TCPChannel [type: server, contact: 127.0.0.1:41194]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:47:01,376-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41194] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174500.517, now=131106-174701.376, channel=TCPChannel [type: server, contact: 127.0.0.1:41194]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:47:01,376-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174500.738, now=131106-174701.576, channel=TCPChannel [type: server, contact: 127.0.0.1:41195]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:47:01,576-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174500.738, now=131106-174701.576, channel=TCPChannel [type: server, contact: 127.0.0.1:41195]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:47:01,576-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174500.738, now=131106-174701.576, channel=TCPChannel [type: server, contact: 127.0.0.1:41195]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:47:01,576-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174500.738, now=131106-174701.576, channel=TCPChannel [type: server, contact: 127.0.0.1:41195]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:47:01,576-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41195] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174500.738, now=131106-174701.576, channel=TCPChannel [type: server, contact: 127.0.0.1:41195]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:47:01,576-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41195] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174500.738, now=131106-174701.576, channel=TCPChannel [type: server, contact: 127.0.0.1:41195]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:47:01,576-0600 INFO ChannelManager Channel already shut down
-2013-11-06 17:47:02,671-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:47:02,671-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:47:02,671-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-06 17:47:02,672-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-Handler(tag: 7487, JOBSTATUS) unregistering (send)
-2013-11-06 17:47:02,672-0600 INFO RequestHandler Handler(tag: 7487, JOBSTATUS) unregistering (send)
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:47:02,672-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-grind.mcs.anl.gov:0 (quality: 78.0) submitting urn:job-2-1383759274789-1383759274790: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:47:02,672-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 78.0) submitting urn:job-2-1383759274789-1383759274790: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-Channel config: null -> {keepalive=null}
-2013-11-06 17:47:02,672-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41199] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:47:02,672-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41199] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-06 17:47:02,672-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-41199, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41199]] binding to NullChannel
-2013-11-06 17:47:02,672-0600 INFO MetaChannel MetaChannel [context: cps-41199, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41199]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41199]
-2013-11-06 17:47:02,673-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41199]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:47:02,673-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:47:02,673-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:47:02,673-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:47:02,673-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:47:02,673-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:47:02,673-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:47:02,673-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:47:02,673-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:47:02,673-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:47:02,673-0600 INFO ChannelManager Channel exception handled
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:47:02,673-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:47:02,674-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41201] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:47:02,674-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41201] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-06 17:47:02,674-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-41201, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41201]] binding to NullChannel
-2013-11-06 17:47:02,674-0600 INFO MetaChannel MetaChannel [context: cps-41201, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41201]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41201]
-2013-11-06 17:47:02,674-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41201]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:47:02,674-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:47:02,674-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:47:02,674-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:47:02,674-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 17:47:02,677-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:47:02,677-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:47:02,677-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41199] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:47:02,677-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41199] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:47:02,677-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41199]
-2013-11-06 17:47:02,677-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41199]
-Channel is closed. Removing.
-2013-11-06 17:47:02,677-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:47:02,677-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:47:02,677-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:47:02,677-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41201] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:47:02,678-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41201] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:47:02,678-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41201]
-2013-11-06 17:47:02,678-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41201]
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:47:02,678-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:47:02,678-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7488, JOBSTATUS) unregistering (send)
-2013-11-06 17:47:02,678-0600 INFO RequestHandler Handler(tag: 7488, JOBSTATUS) unregistering (send)
-2013-11-06 17:47:02,871-0600 DEBUG Cpu Completed
-crush.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:47:02,871-0600 INFO Cpu crush.mcs.anl.gov:0 jobTerminated
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:47:02,871-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:47:02,871-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-Channel config: null -> {keepalive=null}
-2013-11-06 17:47:02,872-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-Handler(tag: 78, JOBSTATUS) unregistering (send)
-2013-11-06 17:47:02,872-0600 INFO RequestHandler Handler(tag: 78, JOBSTATUS) unregistering (send)
-2013-11-06 17:47:02,872-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-TCPChannel [type: server, contact: 127.0.0.1:41200] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:47:02,872-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41200] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:47:02,872-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-null: Channel shut down
-2013-11-06 17:47:02,872-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-41200, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41200]] binding to NullChannel
-2013-11-06 17:47:02,872-0600 INFO MetaChannel MetaChannel [context: cps-41200, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41200]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41200]
-2013-11-06 17:47:02,872-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41200]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:47:02,872-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:47:02,872-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:47:02,872-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:47:02,872-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:47:02,872-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:47:02,873-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:47:02,873-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:47:02,873-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:47:02,873-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:47:02,873-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 17:47:02,878-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:47:02,878-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:47:02,879-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41200] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:47:02,879-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41200] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:47:02,879-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41200]
-2013-11-06 17:47:02,879-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41200]
-No streams
-2013-11-06 17:47:03,701-0600 INFO Multiplexer No streams
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:47:04,690-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:47:04,690-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:47:04,690-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:47:04,690-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-06 17:47:04,690-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-2013-11-06 17:47:04,690-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-Handler(tag: 7489, JOBSTATUS) unregistering (send)
-2013-11-06 17:47:04,690-0600 INFO RequestHandler Handler(tag: 7489, JOBSTATUS) unregistering (send)
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:47:09,788-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:47:13,709-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:47:23,568-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:47:23,717-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:47:33,725-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:47:43,733-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:47:46,262-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:47:53,741-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-06 17:48:00,886-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-06 17:48:03,749-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:48:13,756-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:48:22,736-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:48:23,764-0600 INFO Multiplexer No streams
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174629.793, now=131106-174830.641, channel=TCPChannel [type: server, contact: 127.0.0.1:41196]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:48:30,641-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174629.793, now=131106-174830.641, channel=TCPChannel [type: server, contact: 127.0.0.1:41196]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:48:30,642-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174629.793, now=131106-174830.641, channel=TCPChannel [type: server, contact: 127.0.0.1:41196]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:48:30,642-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174629.793, now=131106-174830.641, channel=TCPChannel [type: server, contact: 127.0.0.1:41196]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:48:30,642-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174629.793, now=131106-174830.641, channel=TCPChannel [type: server, contact: 127.0.0.1:41196]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:48:30,642-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174629.793, now=131106-174830.641, channel=TCPChannel [type: server, contact: 127.0.0.1:41196]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:48:30,642-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174629.793, now=131106-174830.641, channel=TCPChannel [type: server, contact: 127.0.0.1:41196]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:48:30,642-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174629.793, now=131106-174830.641, channel=TCPChannel [type: server, contact: 127.0.0.1:41196]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:48:30,642-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174629.793, now=131106-174830.641, channel=TCPChannel [type: server, contact: 127.0.0.1:41196]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:48:30,642-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174629.793, now=131106-174830.641, channel=TCPChannel [type: server, contact: 127.0.0.1:41196]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:48:30,642-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41196] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174629.793, now=131106-174830.641, channel=TCPChannel [type: server, contact: 127.0.0.1:41196]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:48:30,642-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41196] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174629.793, now=131106-174830.641, channel=TCPChannel [type: server, contact: 127.0.0.1:41196]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:48:30,642-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174629.995, now=131106-174830.852, channel=TCPChannel [type: server, contact: 127.0.0.1:41197]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:48:30,852-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174629.995, now=131106-174830.852, channel=TCPChannel [type: server, contact: 127.0.0.1:41197]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:48:30,853-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174629.995, now=131106-174830.852, channel=TCPChannel [type: server, contact: 127.0.0.1:41197]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:48:30,853-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174629.995, now=131106-174830.852, channel=TCPChannel [type: server, contact: 127.0.0.1:41197]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:48:30,853-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174629.995, now=131106-174830.852, channel=TCPChannel [type: server, contact: 127.0.0.1:41197]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:48:30,853-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174629.995, now=131106-174830.852, channel=TCPChannel [type: server, contact: 127.0.0.1:41197]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:48:30,853-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174629.995, now=131106-174830.852, channel=TCPChannel [type: server, contact: 127.0.0.1:41197]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:48:30,853-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174629.995, now=131106-174830.852, channel=TCPChannel [type: server, contact: 127.0.0.1:41197]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:48:30,853-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174629.995, now=131106-174830.852, channel=TCPChannel [type: server, contact: 127.0.0.1:41197]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:48:30,853-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174629.995, now=131106-174830.852, channel=TCPChannel [type: server, contact: 127.0.0.1:41197]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:48:30,853-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41197] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174629.995, now=131106-174830.852, channel=TCPChannel [type: server, contact: 127.0.0.1:41197]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:48:30,853-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41197] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174629.995, now=131106-174830.852, channel=TCPChannel [type: server, contact: 127.0.0.1:41197]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:48:30,853-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174630.197, now=131106-174831.055, channel=TCPChannel [type: server, contact: 127.0.0.1:41198]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:48:31,055-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174630.197, now=131106-174831.055, channel=TCPChannel [type: server, contact: 127.0.0.1:41198]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:48:31,055-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174630.197, now=131106-174831.055, channel=TCPChannel [type: server, contact: 127.0.0.1:41198]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:48:31,055-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174630.197, now=131106-174831.055, channel=TCPChannel [type: server, contact: 127.0.0.1:41198]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:48:31,056-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41198] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174630.197, now=131106-174831.055, channel=TCPChannel [type: server, contact: 127.0.0.1:41198]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:48:31,056-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41198] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174630.197, now=131106-174831.055, channel=TCPChannel [type: server, contact: 127.0.0.1:41198]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:48:31,056-0600 INFO ChannelManager Channel already shut down
-No streams
-2013-11-06 17:48:33,772-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-06 17:48:38,204-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-06 17:48:43,780-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:48:53,788-0600 INFO Multiplexer No streams
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:48:54,867-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41202] for addition
-2013-11-06 17:48:54,867-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41202] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41202]
-2013-11-06 17:48:54,867-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41202]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f8d
-2013-11-06 17:48:54,892-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f8d
-MetaChannel [context: cps-41202, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41202]
-2013-11-06 17:48:54,892-0600 INFO MetaChannel MetaChannel [context: cps-41202, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41202]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:48:54,892-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:48:54,932-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 17:48:54,932-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 17:48:55,013-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 17:48:55,017-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 79.0) submitting urn:job-0-1383759274792-1383759274793: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:48:55,017-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 79.0) submitting urn:job-0-1383759274792-1383759274793: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:48:55,017-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 17:48:55,017-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-06 17:48:55,023-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7490, JOBSTATUS) unregistering (send)
-2013-11-06 17:48:55,023-0600 INFO RequestHandler Handler(tag: 7490, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:48:55,072-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41203] for addition
-2013-11-06 17:48:55,072-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41203] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41203]
-2013-11-06 17:48:55,072-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41203]
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f8c
-2013-11-06 17:48:55,094-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f8c
-MetaChannel [context: cps-41203, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41203]
-2013-11-06 17:48:55,094-0600 INFO MetaChannel MetaChannel [context: cps-41203, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41203]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:48:55,094-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:48:55,134-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 17:48:55,215-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 17:48:55,218-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-crush.mcs.anl.gov:0 (quality: 24.0) submitting urn:job-1-1383759274795-1383759274796: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:48:55,218-0600 INFO Cpu crush.mcs.anl.gov:0 (quality: 24.0) submitting urn:job-1-1383759274795-1383759274796: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:48:55,225-0600 DEBUG Cpu Active workerid=crush.mcs.anl.gov:000000
-Handler(tag: 79, JOBSTATUS) unregistering (send)
-2013-11-06 17:48:55,225-0600 INFO RequestHandler Handler(tag: 79, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:48:55,272-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41204] for addition
-2013-11-06 17:48:55,272-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41204] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41204]
-2013-11-06 17:48:55,272-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41204]
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f8b
-2013-11-06 17:48:55,295-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f8b
-MetaChannel [context: cps-41204, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41204]
-2013-11-06 17:48:55,295-0600 INFO MetaChannel MetaChannel [context: cps-41204, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41204]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:48:55,295-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:48:55,336-0600 INFO RequestHandler Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-06 17:48:55,416-0600 INFO RequestHandler Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-06 17:48:57,042-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:48:57,042-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:48:57,042-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:48:57,042-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Channel config: null -> {keepalive=null}
-2013-11-06 17:48:57,042-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-Handler(tag: 7491, JOBSTATUS) unregistering (send)
-2013-11-06 17:48:57,043-0600 INFO RequestHandler Handler(tag: 7491, JOBSTATUS) unregistering (send)
-TCPChannel [type: server, contact: 127.0.0.1:41202] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:48:57,043-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41202] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:48:57,043-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-null: Channel shut down
-2013-11-06 17:48:57,043-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-41202, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41202]] binding to NullChannel
-2013-11-06 17:48:57,043-0600 INFO MetaChannel MetaChannel [context: cps-41202, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41202]] binding to NullChannel
-grind.mcs.anl.gov:0 (quality: 80.0) submitting urn:job-2-1383759274798-1383759274799: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:48:57,043-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 80.0) submitting urn:job-2-1383759274798-1383759274799: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41202]
-2013-11-06 17:48:57,043-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41202]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:48:57,043-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:48:57,044-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:48:57,044-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:48:57,044-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:48:57,044-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:48:57,044-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:48:57,044-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:48:57,044-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:48:57,044-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:48:57,044-0600 INFO ChannelManager Channel exception handled
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:48:57,044-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:48:57,044-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41204] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:48:57,044-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41204] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-06 17:48:57,045-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-41204, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41204]] binding to NullChannel
-2013-11-06 17:48:57,045-0600 INFO MetaChannel MetaChannel [context: cps-41204, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41204]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41204]
-2013-11-06 17:48:57,045-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41204]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:48:57,045-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:48:57,045-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:48:57,045-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:48:57,045-0600 INFO ChannelManager Channel exception handled
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:48:57,049-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:48:57,050-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:48:57,051-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7492, JOBSTATUS) unregistering (send)
-2013-11-06 17:48:57,051-0600 INFO RequestHandler Handler(tag: 7492, JOBSTATUS) unregistering (send)
-Channel is closed. Removing.
-2013-11-06 17:48:57,061-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:48:57,061-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:48:57,061-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41204] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:48:57,061-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41204] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:48:57,061-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41204]
-2013-11-06 17:48:57,061-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41204]
-Channel is closed. Removing.
-2013-11-06 17:48:57,062-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:48:57,062-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:48:57,062-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41202] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:48:57,062-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41202] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:48:57,062-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41202]
-2013-11-06 17:48:57,062-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41202]
-2013-11-06 17:48:57,242-0600 DEBUG Cpu Completed
-crush.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:48:57,242-0600 INFO Cpu crush.mcs.anl.gov:0 jobTerminated
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:48:57,242-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:48:57,242-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-Channel config: null -> {keepalive=null}
-2013-11-06 17:48:57,243-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-Handler(tag: 80, JOBSTATUS) unregistering (send)
-2013-11-06 17:48:57,243-0600 INFO RequestHandler Handler(tag: 80, JOBSTATUS) unregistering (send)
-2013-11-06 17:48:57,243-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-TCPChannel [type: server, contact: 127.0.0.1:41203] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:48:57,243-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41203] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:48:57,243-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-null: Channel shut down
-2013-11-06 17:48:57,243-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-41203, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41203]] binding to NullChannel
-2013-11-06 17:48:57,243-0600 INFO MetaChannel MetaChannel [context: cps-41203, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41203]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41203]
-2013-11-06 17:48:57,243-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41203]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:48:57,243-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:48:57,243-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:48:57,243-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:48:57,243-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:48:57,243-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:48:57,243-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:48:57,244-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:48:57,244-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:48:57,244-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:48:57,244-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 17:48:57,262-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:48:57,262-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:48:57,263-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41203] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:48:57,263-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41203] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:48:57,263-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41203]
-2013-11-06 17:48:57,263-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41203]
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:48:59,094-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:48:59,094-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:48:59,094-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:48:59,094-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-06 17:48:59,094-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-2013-11-06 17:48:59,094-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-Handler(tag: 7493, JOBSTATUS) unregistering (send)
-2013-11-06 17:48:59,094-0600 INFO RequestHandler Handler(tag: 7493, JOBSTATUS) unregistering (send)
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:48:59,210-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174700.605, now=131106-174901.449, channel=TCPChannel [type: server, contact: 127.0.0.1:41199]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:49:01,449-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174700.605, now=131106-174901.449, channel=TCPChannel [type: server, contact: 127.0.0.1:41199]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:49:01,450-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174700.605, now=131106-174901.449, channel=TCPChannel [type: server, contact: 127.0.0.1:41199]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:49:01,450-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174700.605, now=131106-174901.449, channel=TCPChannel [type: server, contact: 127.0.0.1:41199]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:49:01,450-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174700.605, now=131106-174901.449, channel=TCPChannel [type: server, contact: 127.0.0.1:41199]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:49:01,450-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174700.605, now=131106-174901.449, channel=TCPChannel [type: server, contact: 127.0.0.1:41199]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:49:01,450-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174700.605, now=131106-174901.449, channel=TCPChannel [type: server, contact: 127.0.0.1:41199]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:49:01,450-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174700.605, now=131106-174901.449, channel=TCPChannel [type: server, contact: 127.0.0.1:41199]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:49:01,450-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174700.605, now=131106-174901.449, channel=TCPChannel [type: server, contact: 127.0.0.1:41199]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:49:01,450-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174700.605, now=131106-174901.449, channel=TCPChannel [type: server, contact: 127.0.0.1:41199]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:49:01,450-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41199] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174700.605, now=131106-174901.449, channel=TCPChannel [type: server, contact: 127.0.0.1:41199]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:49:01,450-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41199] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174700.605, now=131106-174901.449, channel=TCPChannel [type: server, contact: 127.0.0.1:41199]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:49:01,450-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174700.806, now=131106-174901.664, channel=TCPChannel [type: server, contact: 127.0.0.1:41200]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:49:01,664-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174700.806, now=131106-174901.664, channel=TCPChannel [type: server, contact: 127.0.0.1:41200]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:49:01,664-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174700.806, now=131106-174901.664, channel=TCPChannel [type: server, contact: 127.0.0.1:41200]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:49:01,664-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174700.806, now=131106-174901.664, channel=TCPChannel [type: server, contact: 127.0.0.1:41200]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:49:01,665-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174700.806, now=131106-174901.664, channel=TCPChannel [type: server, contact: 127.0.0.1:41200]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:49:01,665-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174700.806, now=131106-174901.664, channel=TCPChannel [type: server, contact: 127.0.0.1:41200]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:49:01,665-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174700.806, now=131106-174901.664, channel=TCPChannel [type: server, contact: 127.0.0.1:41200]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:49:01,665-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174700.806, now=131106-174901.664, channel=TCPChannel [type: server, contact: 127.0.0.1:41200]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:49:01,665-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174700.806, now=131106-174901.664, channel=TCPChannel [type: server, contact: 127.0.0.1:41200]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:49:01,665-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174700.806, now=131106-174901.664, channel=TCPChannel [type: server, contact: 127.0.0.1:41200]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:49:01,665-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41200] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174700.806, now=131106-174901.664, channel=TCPChannel [type: server, contact: 127.0.0.1:41200]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:49:01,665-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41200] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174700.806, now=131106-174901.664, channel=TCPChannel [type: server, contact: 127.0.0.1:41200]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:49:01,665-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174701.012, now=131106-174901.864, channel=TCPChannel [type: server, contact: 127.0.0.1:41201]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:49:01,864-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174701.012, now=131106-174901.864, channel=TCPChannel [type: server, contact: 127.0.0.1:41201]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:49:01,864-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174701.012, now=131106-174901.864, channel=TCPChannel [type: server, contact: 127.0.0.1:41201]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:49:01,864-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174701.012, now=131106-174901.864, channel=TCPChannel [type: server, contact: 127.0.0.1:41201]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:49:01,864-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41201] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174701.012, now=131106-174901.864, channel=TCPChannel [type: server, contact: 127.0.0.1:41201]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:49:01,864-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41201] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174701.012, now=131106-174901.864, channel=TCPChannel [type: server, contact: 127.0.0.1:41201]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:49:01,864-0600 INFO ChannelManager Channel already shut down
-No streams
-2013-11-06 17:49:03,801-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:49:13,809-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:49:15,522-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:49:23,817-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:49:33,826-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:49:35,684-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:49:43,834-0600 INFO Multiplexer No streams
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:49:44,307-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41205] for addition
-2013-11-06 17:49:44,307-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41205] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41205]
-2013-11-06 17:49:44,307-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41205]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f8a
-2013-11-06 17:49:44,336-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f8a
-MetaChannel [context: cps-41205, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41205]
-2013-11-06 17:49:44,337-0600 INFO MetaChannel MetaChannel [context: cps-41205, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41205]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:49:44,337-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:49:44,377-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 17:49:44,377-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 17:49:44,458-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 17:49:44,496-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 81.0) submitting urn:job-0-1383759274801-1383759274802: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:49:44,496-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 81.0) submitting urn:job-0-1383759274801-1383759274802: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:49:44,497-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 17:49:44,497-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:49:44,516-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41206] for addition
-2013-11-06 17:49:44,516-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41206] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41206]
-2013-11-06 17:49:44,516-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41206]
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f89
-2013-11-06 17:49:44,538-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f89
-MetaChannel [context: cps-41206, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41206]
-2013-11-06 17:49:44,538-0600 INFO MetaChannel MetaChannel [context: cps-41206, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41206]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:49:44,538-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:49:44,579-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 17:49:44,659-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 17:49:44,697-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-crush.mcs.anl.gov:0 (quality: 25.0) submitting urn:job-1-1383759274804-1383759274805: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:49:44,697-0600 INFO Cpu crush.mcs.anl.gov:0 (quality: 25.0) submitting urn:job-1-1383759274804-1383759274805: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:49:44,711-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7494, JOBSTATUS) unregistering (send)
-2013-11-06 17:49:44,711-0600 INFO RequestHandler Handler(tag: 7494, JOBSTATUS) unregistering (send)
-2013-11-06 17:49:44,717-0600 DEBUG Cpu Active workerid=crush.mcs.anl.gov:000000
-Handler(tag: 81, JOBSTATUS) unregistering (send)
-2013-11-06 17:49:44,717-0600 INFO RequestHandler Handler(tag: 81, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:49:44,720-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41207] for addition
-2013-11-06 17:49:44,721-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41207] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41207]
-2013-11-06 17:49:44,721-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41207]
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f88
-2013-11-06 17:49:44,760-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f88
-MetaChannel [context: cps-41207, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41207]
-2013-11-06 17:49:44,760-0600 INFO MetaChannel MetaChannel [context: cps-41207, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41207]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:49:44,760-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:49:44,780-0600 INFO RequestHandler Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-06 17:49:44,861-0600 INFO RequestHandler Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-06 17:49:46,721-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:49:46,722-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:49:46,722-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:49:46,722-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Channel config: null -> {keepalive=null}
-2013-11-06 17:49:46,722-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41205] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:49:46,722-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41205] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:49:46,722-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-null: Channel shut down
-2013-11-06 17:49:46,722-0600 INFO AbstractTCPChannel null: Channel shut down
-Handler(tag: 7495, JOBSTATUS) unregistering (send)
-2013-11-06 17:49:46,722-0600 INFO RequestHandler Handler(tag: 7495, JOBSTATUS) unregistering (send)
-grind.mcs.anl.gov:0 (quality: 82.0) submitting urn:job-2-1383759274807-1383759274808: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:49:46,722-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 82.0) submitting urn:job-2-1383759274807-1383759274808: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-MetaChannel [context: cps-41205, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41205]] binding to NullChannel
-2013-11-06 17:49:46,722-0600 INFO MetaChannel MetaChannel [context: cps-41205, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41205]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41205]
-2013-11-06 17:49:46,723-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41205]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:49:46,723-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:49:46,723-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:49:46,723-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:49:46,723-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:49:46,723-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:49:46,723-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:49:46,723-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:49:46,723-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:49:46,723-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:49:46,724-0600 INFO ChannelManager Channel exception handled
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:49:46,724-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:49:46,724-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41207] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:49:46,724-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41207] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-06 17:49:46,724-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-41207, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41207]] binding to NullChannel
-2013-11-06 17:49:46,724-0600 INFO MetaChannel MetaChannel [context: cps-41207, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41207]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41207]
-2013-11-06 17:49:46,724-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41207]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:49:46,724-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:49:46,724-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:49:46,724-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:49:46,724-0600 INFO ChannelManager Channel exception handled
-2013-11-06 17:49:46,727-0600 DEBUG Cpu Completed
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:49:46,727-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel is closed. Removing.
-2013-11-06 17:49:46,727-0600 INFO Multiplexer Channel is closed. Removing.
-crush.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:49:46,727-0600 INFO Cpu crush.mcs.anl.gov:0 jobTerminated
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:49:46,727-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:49:46,727-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-Channel config: null -> {keepalive=null}
-2013-11-06 17:49:46,727-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-06 17:49:46,727-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-TCPChannel [type: server, contact: 127.0.0.1:41207] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:49:46,727-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41207] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:49:46,727-0600 INFO ChannelManager Channel already shut down
-TCPChannel [type: server, contact: 127.0.0.1:41206] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:49:46,727-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41206] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41207]
-2013-11-06 17:49:46,727-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41207]
-Handler(tag: 82, JOBSTATUS) unregistering (send)
-2013-11-06 17:49:46,727-0600 INFO RequestHandler Handler(tag: 82, JOBSTATUS) unregistering (send)
-2013-11-06 17:49:46,727-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 17:49:46,728-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-Channel is closed. Removing.
-2013-11-06 17:49:46,728-0600 INFO Multiplexer Channel is closed. Removing.
-null: Channel shut down
-2013-11-06 17:49:46,728-0600 INFO AbstractTCPChannel null: Channel shut down
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:49:46,728-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-MetaChannel [context: cps-41206, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41206]] binding to NullChannel
-2013-11-06 17:49:46,728-0600 INFO MetaChannel MetaChannel [context: cps-41206, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41206]] binding to NullChannel
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:49:46,728-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41206]
-2013-11-06 17:49:46,728-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41206]
-Channel config: null -> {keepalive=null}
-2013-11-06 17:49:46,728-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:49:46,728-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-TCPChannel [type: server, contact: 127.0.0.1:41205] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:49:46,729-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41205] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:49:46,729-0600 INFO ChannelManager Channel already shut down
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:49:46,729-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:49:46,729-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:49:46,729-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Handler(tag: 7496, JOBSTATUS) unregistering (send)
-2013-11-06 17:49:46,729-0600 INFO RequestHandler Handler(tag: 7496, JOBSTATUS) unregistering (send)
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41205]
-2013-11-06 17:49:46,729-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41205]
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:49:46,729-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:49:46,729-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:49:46,729-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:49:46,729-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:49:46,730-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:49:46,730-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:49:46,730-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:49:46,730-0600 INFO ChannelManager Channel exception handled
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:49:48,739-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:49:48,739-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:49:48,739-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:49:48,739-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-06 17:49:48,739-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-2013-11-06 17:49:48,739-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-Handler(tag: 7497, JOBSTATUS) unregistering (send)
-2013-11-06 17:49:48,739-0600 INFO RequestHandler Handler(tag: 7497, JOBSTATUS) unregistering (send)
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:49:52,840-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:49:53,849-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:50:03,857-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:50:12,158-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:50:13,866-0600 INFO Multiplexer No streams
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:50:18,949-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41208] for addition
-2013-11-06 17:50:18,950-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41208] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41208]
-2013-11-06 17:50:18,950-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41208]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f87
-2013-11-06 17:50:18,981-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f87
-MetaChannel [context: cps-41208, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41208]
-2013-11-06 17:50:18,982-0600 INFO MetaChannel MetaChannel [context: cps-41208, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41208]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:50:18,982-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:50:19,022-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 17:50:19,022-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 17:50:19,103-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 17:50:19,123-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 83.0) submitting urn:job-0-1383759274810-1383759274811: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:50:19,124-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 83.0) submitting urn:job-0-1383759274810-1383759274811: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:50:19,124-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 17:50:19,124-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-06 17:50:19,130-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7498, JOBSTATUS) unregistering (send)
-2013-11-06 17:50:19,130-0600 INFO RequestHandler Handler(tag: 7498, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:50:19,160-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41209] for addition
-2013-11-06 17:50:19,160-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41209] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41209]
-2013-11-06 17:50:19,160-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41209]
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f86
-2013-11-06 17:50:19,183-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f86
-MetaChannel [context: cps-41209, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41209]
-2013-11-06 17:50:19,183-0600 INFO MetaChannel MetaChannel [context: cps-41209, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41209]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:50:19,183-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:50:19,224-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 17:50:19,304-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 17:50:19,324-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-crush.mcs.anl.gov:0 (quality: 26.0) submitting urn:job-1-1383759274813-1383759274814: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:50:19,324-0600 INFO Cpu crush.mcs.anl.gov:0 (quality: 26.0) submitting urn:job-1-1383759274813-1383759274814: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:50:19,330-0600 DEBUG Cpu Active workerid=crush.mcs.anl.gov:000000
-Handler(tag: 83, JOBSTATUS) unregistering (send)
-2013-11-06 17:50:19,330-0600 INFO RequestHandler Handler(tag: 83, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:50:19,364-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41210] for addition
-2013-11-06 17:50:19,364-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41210] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41210]
-2013-11-06 17:50:19,364-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41210]
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f85
-2013-11-06 17:50:19,385-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f85
-MetaChannel [context: cps-41210, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41210]
-2013-11-06 17:50:19,385-0600 INFO MetaChannel MetaChannel [context: cps-41210, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41210]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:50:19,385-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:50:19,426-0600 INFO RequestHandler Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-06 17:50:19,506-0600 INFO RequestHandler Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-06 17:50:21,232-0600 DEBUG Cpu Completed
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:50:21,232-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:50:21,232-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-Channel config: null -> {keepalive=null}
-2013-11-06 17:50:21,232-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-06 17:50:21,232-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-TCPChannel [type: server, contact: 127.0.0.1:41208] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:50:21,232-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41208] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 7499, JOBSTATUS) unregistering (send)
-2013-11-06 17:50:21,232-0600 INFO RequestHandler Handler(tag: 7499, JOBSTATUS) unregistering (send)
-2013-11-06 17:50:21,232-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 84.0) submitting urn:job-2-1383759274816-1383759274817: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:50:21,233-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 84.0) submitting urn:job-2-1383759274816-1383759274817: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-null: Channel shut down
-2013-11-06 17:50:21,233-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-41208, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41208]] binding to NullChannel
-2013-11-06 17:50:21,233-0600 INFO MetaChannel MetaChannel [context: cps-41208, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41208]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41208]
-2013-11-06 17:50:21,233-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41208]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:50:21,233-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:50:21,233-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:50:21,234-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:50:21,234-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:50:21,234-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:50:21,234-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:50:21,234-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:50:21,234-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:50:21,234-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:50:21,234-0600 INFO ChannelManager Channel exception handled
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:50:21,234-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:50:21,234-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41210] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:50:21,234-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41210] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-06 17:50:21,235-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-41210, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41210]] binding to NullChannel
-2013-11-06 17:50:21,235-0600 INFO MetaChannel MetaChannel [context: cps-41210, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41210]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41210]
-2013-11-06 17:50:21,235-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41210]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:50:21,235-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:50:21,235-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:50:21,235-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:50:21,235-0600 INFO ChannelManager Channel exception handled
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:50:21,237-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:50:21,239-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:50:21,239-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7500, JOBSTATUS) unregistering (send)
-2013-11-06 17:50:21,239-0600 INFO RequestHandler Handler(tag: 7500, JOBSTATUS) unregistering (send)
-Channel is closed. Removing.
-2013-11-06 17:50:21,252-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:50:21,252-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:50:21,252-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41210] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:50:21,252-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41210] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:50:21,252-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41210]
-2013-11-06 17:50:21,252-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41210]
-Channel is closed. Removing.
-2013-11-06 17:50:21,252-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:50:21,252-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:50:21,252-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41208] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:50:21,252-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41208] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:50:21,252-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41208]
-2013-11-06 17:50:21,252-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41208]
-2013-11-06 17:50:21,348-0600 DEBUG Cpu Completed
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:50:21,348-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-crush.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:50:21,348-0600 INFO Cpu crush.mcs.anl.gov:0 jobTerminated
-Channel config: null -> {keepalive=null}
-2013-11-06 17:50:21,348-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-06 17:50:21,348-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-TCPChannel [type: server, contact: 127.0.0.1:41209] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:50:21,348-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41209] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 84, JOBSTATUS) unregistering (send)
-2013-11-06 17:50:21,348-0600 INFO RequestHandler Handler(tag: 84, JOBSTATUS) unregistering (send)
-2013-11-06 17:50:21,348-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 17:50:21,349-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-null: Channel shut down
-2013-11-06 17:50:21,349-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-41209, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41209]] binding to NullChannel
-2013-11-06 17:50:21,349-0600 INFO MetaChannel MetaChannel [context: cps-41209, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41209]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41209]
-2013-11-06 17:50:21,349-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41209]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:50:21,349-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:50:21,349-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:50:21,349-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:50:21,349-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:50:21,349-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:50:21,349-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:50:21,349-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:50:21,349-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:50:21,350-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:50:21,350-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 17:50:21,353-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:50:21,353-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:50:21,353-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41209] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:50:21,353-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41209] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:50:21,353-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41209]
-2013-11-06 17:50:21,353-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41209]
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:50:23,249-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:50:23,249-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:50:23,249-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:50:23,249-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-06 17:50:23,249-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-2013-11-06 17:50:23,249-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-Handler(tag: 7501, JOBSTATUS) unregistering (send)
-2013-11-06 17:50:23,249-0600 INFO RequestHandler Handler(tag: 7501, JOBSTATUS) unregistering (send)
-No streams
-2013-11-06 17:50:23,880-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:50:30,158-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:50:33,888-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:50:43,896-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:50:48,632-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:50:53,904-0600 INFO Multiplexer No streams
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:50:55,226-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41211] for addition
-2013-11-06 17:50:55,226-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41211] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41211]
-2013-11-06 17:50:55,226-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41211]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f84
-2013-11-06 17:50:55,252-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f84
-MetaChannel [context: cps-41211, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41211]
-2013-11-06 17:50:55,252-0600 INFO MetaChannel MetaChannel [context: cps-41211, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41211]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:50:55,252-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:50:55,293-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 17:50:55,293-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 17:50:55,393-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 17:50:55,436-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 85.0) submitting urn:job-0-1383759274819-1383759274820: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:50:55,436-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 85.0) submitting urn:job-0-1383759274819-1383759274820: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:50:55,437-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 17:50:55,437-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:50:55,439-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41212] for addition
-2013-11-06 17:50:55,440-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41212] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41212]
-2013-11-06 17:50:55,440-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41212]
-2013-11-06 17:50:55,443-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7502, JOBSTATUS) unregistering (send)
-2013-11-06 17:50:55,443-0600 INFO RequestHandler Handler(tag: 7502, JOBSTATUS) unregistering (send)
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f83
-2013-11-06 17:50:55,474-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f83
-MetaChannel [context: cps-41212, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41212]
-2013-11-06 17:50:55,474-0600 INFO MetaChannel MetaChannel [context: cps-41212, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41212]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:50:55,474-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:50:55,514-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 17:50:55,595-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 17:50:55,637-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-crush.mcs.anl.gov:0 (quality: 27.0) submitting urn:job-1-1383759274822-1383759274823: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:50:55,637-0600 INFO Cpu crush.mcs.anl.gov:0 (quality: 27.0) submitting urn:job-1-1383759274822-1383759274823: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:50:55,643-0600 DEBUG Cpu Active workerid=crush.mcs.anl.gov:000000
-Handler(tag: 85, JOBSTATUS) unregistering (send)
-2013-11-06 17:50:55,643-0600 INFO RequestHandler Handler(tag: 85, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 17:50:55,652-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41213] for addition
-2013-11-06 17:50:55,652-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41213] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41213]
-2013-11-06 17:50:55,652-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41213]
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f82
-2013-11-06 17:50:55,675-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f82
-MetaChannel [context: cps-41213, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41213]
-2013-11-06 17:50:55,675-0600 INFO MetaChannel MetaChannel [context: cps-41213, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41213]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-06 17:50:55,676-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-06 17:50:55,716-0600 INFO RequestHandler Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-06 17:50:55,796-0600 INFO RequestHandler Handler(tag: 42566, SUBMITJOB) unregistering (send)
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174855.013, now=131106-175055.867, channel=TCPChannel [type: server, contact: 127.0.0.1:41202]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:50:55,867-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174855.013, now=131106-175055.867, channel=TCPChannel [type: server, contact: 127.0.0.1:41202]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:50:55,868-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174855.013, now=131106-175055.867, channel=TCPChannel [type: server, contact: 127.0.0.1:41202]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:50:55,868-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174855.013, now=131106-175055.867, channel=TCPChannel [type: server, contact: 127.0.0.1:41202]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:50:55,868-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174855.013, now=131106-175055.867, channel=TCPChannel [type: server, contact: 127.0.0.1:41202]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:50:55,868-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174855.013, now=131106-175055.867, channel=TCPChannel [type: server, contact: 127.0.0.1:41202]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:50:55,868-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174855.013, now=131106-175055.867, channel=TCPChannel [type: server, contact: 127.0.0.1:41202]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:50:55,868-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174855.013, now=131106-175055.867, channel=TCPChannel [type: server, contact: 127.0.0.1:41202]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:50:55,868-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174855.013, now=131106-175055.867, channel=TCPChannel [type: server, contact: 127.0.0.1:41202]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:50:55,868-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174855.013, now=131106-175055.867, channel=TCPChannel [type: server, contact: 127.0.0.1:41202]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:50:55,868-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41202] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174855.013, now=131106-175055.867, channel=TCPChannel [type: server, contact: 127.0.0.1:41202]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:50:55,868-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41202] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174855.013, now=131106-175055.867, channel=TCPChannel [type: server, contact: 127.0.0.1:41202]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:50:55,868-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174855.214, now=131106-175056.072, channel=TCPChannel [type: server, contact: 127.0.0.1:41203]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:50:56,072-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174855.214, now=131106-175056.072, channel=TCPChannel [type: server, contact: 127.0.0.1:41203]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:50:56,073-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174855.214, now=131106-175056.072, channel=TCPChannel [type: server, contact: 127.0.0.1:41203]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:50:56,073-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174855.214, now=131106-175056.072, channel=TCPChannel [type: server, contact: 127.0.0.1:41203]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:50:56,073-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174855.214, now=131106-175056.072, channel=TCPChannel [type: server, contact: 127.0.0.1:41203]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:50:56,073-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174855.214, now=131106-175056.072, channel=TCPChannel [type: server, contact: 127.0.0.1:41203]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:50:56,073-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174855.214, now=131106-175056.072, channel=TCPChannel [type: server, contact: 127.0.0.1:41203]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:50:56,073-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174855.214, now=131106-175056.072, channel=TCPChannel [type: server, contact: 127.0.0.1:41203]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:50:56,073-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174855.214, now=131106-175056.072, channel=TCPChannel [type: server, contact: 127.0.0.1:41203]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:50:56,073-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174855.214, now=131106-175056.072, channel=TCPChannel [type: server, contact: 127.0.0.1:41203]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:50:56,073-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41203] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174855.214, now=131106-175056.072, channel=TCPChannel [type: server, contact: 127.0.0.1:41203]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:50:56,073-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41203] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174855.214, now=131106-175056.072, channel=TCPChannel [type: server, contact: 127.0.0.1:41203]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:50:56,073-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174855.416, now=131106-175056.272, channel=TCPChannel [type: server, contact: 127.0.0.1:41204]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:50:56,272-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174855.416, now=131106-175056.272, channel=TCPChannel [type: server, contact: 127.0.0.1:41204]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:50:56,272-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174855.416, now=131106-175056.272, channel=TCPChannel [type: server, contact: 127.0.0.1:41204]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:50:56,272-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174855.416, now=131106-175056.272, channel=TCPChannel [type: server, contact: 127.0.0.1:41204]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:50:56,272-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41204] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174855.416, now=131106-175056.272, channel=TCPChannel [type: server, contact: 127.0.0.1:41204]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:50:56,272-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41204] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174855.416, now=131106-175056.272, channel=TCPChannel [type: server, contact: 127.0.0.1:41204]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:50:56,272-0600 INFO ChannelManager Channel already shut down
-2013-11-06 17:50:57,455-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:50:57,455-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:50:57,455-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:50:57,455-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Channel config: null -> {keepalive=null}
-2013-11-06 17:50:57,455-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41211] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:50:57,455-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41211] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:50:57,455-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-null: Channel shut down
-2013-11-06 17:50:57,456-0600 INFO AbstractTCPChannel null: Channel shut down
-Handler(tag: 7503, JOBSTATUS) unregistering (send)
-2013-11-06 17:50:57,455-0600 INFO RequestHandler Handler(tag: 7503, JOBSTATUS) unregistering (send)
-MetaChannel [context: cps-41211, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41211]] binding to NullChannel
-2013-11-06 17:50:57,456-0600 INFO MetaChannel MetaChannel [context: cps-41211, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41211]] binding to NullChannel
-grind.mcs.anl.gov:0 (quality: 86.0) submitting urn:job-2-1383759274825-1383759274826: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 17:50:57,456-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 86.0) submitting urn:job-2-1383759274825-1383759274826: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41211]
-2013-11-06 17:50:57,456-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41211]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:50:57,456-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:50:57,456-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:50:57,456-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:50:57,456-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:50:57,456-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:50:57,457-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:50:57,457-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:50:57,457-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:50:57,457-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:50:57,457-0600 INFO ChannelManager Channel exception handled
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:50:57,457-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:50:57,457-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41213] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:50:57,457-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41213] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-06 17:50:57,457-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-41213, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41213]] binding to NullChannel
-2013-11-06 17:50:57,457-0600 INFO MetaChannel MetaChannel [context: cps-41213, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41213]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41213]
-2013-11-06 17:50:57,457-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41213]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:50:57,457-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:50:57,457-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:50:57,457-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:50:57,458-0600 INFO ChannelManager Channel exception handled
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:50:57,461-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:50:57,461-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:50:57,462-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7504, JOBSTATUS) unregistering (send)
-2013-11-06 17:50:57,462-0600 INFO RequestHandler Handler(tag: 7504, JOBSTATUS) unregistering (send)
-Channel is closed. Removing.
-2013-11-06 17:50:57,463-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:50:57,463-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:50:57,463-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41211] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:50:57,463-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41211] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:50:57,463-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41211]
-2013-11-06 17:50:57,463-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41211]
-Channel is closed. Removing.
-2013-11-06 17:50:57,463-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:50:57,464-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:50:57,464-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41213] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:50:57,464-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41213] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:50:57,464-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41213]
-2013-11-06 17:50:57,464-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41213]
-2013-11-06 17:50:57,655-0600 DEBUG Cpu Completed
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:50:57,655-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-crush.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:50:57,655-0600 INFO Cpu crush.mcs.anl.gov:0 jobTerminated
-Channel config: null -> {keepalive=null}
-2013-11-06 17:50:57,655-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-06 17:50:57,655-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-TCPChannel [type: server, contact: 127.0.0.1:41212] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:50:57,655-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41212] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:50:57,655-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 17:50:57,655-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-null: Channel shut down
-2013-11-06 17:50:57,655-0600 INFO AbstractTCPChannel null: Channel shut down
-Handler(tag: 86, JOBSTATUS) unregistering (send)
-2013-11-06 17:50:57,655-0600 INFO RequestHandler Handler(tag: 86, JOBSTATUS) unregistering (send)
-MetaChannel [context: cps-41212, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41212]] binding to NullChannel
-2013-11-06 17:50:57,655-0600 INFO MetaChannel MetaChannel [context: cps-41212, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41212]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41212]
-2013-11-06 17:50:57,656-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41212]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:50:57,656-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:50:57,656-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:50:57,656-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:50:57,656-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:50:57,656-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:50:57,656-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:50:57,656-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:50:57,656-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 17:50:57,656-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 17:50:57,656-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 17:50:57,664-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:50:57,664-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:50:57,665-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41212] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 17:50:57,665-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41212] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 17:50:57,665-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41212]
-2013-11-06 17:50:57,665-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41212]
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:50:59,472-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 17:50:59,473-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:50:59,473-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 17:50:59,473-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-06 17:50:59,473-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7505, JOBSTATUS) unregistering (send)
-2013-11-06 17:50:59,473-0600 INFO RequestHandler Handler(tag: 7505, JOBSTATUS) unregistering (send)
-2013-11-06 17:50:59,473-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-No streams
-2013-11-06 17:51:03,922-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:51:07,476-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:51:13,930-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:51:23,937-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 17:51:25,106-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 17:51:33,945-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:51:43,953-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:51:44,794-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174944.457, now=131106-175145.307, channel=TCPChannel [type: server, contact: 127.0.0.1:41205]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:51:45,307-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174944.457, now=131106-175145.307, channel=TCPChannel [type: server, contact: 127.0.0.1:41205]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:51:45,307-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174944.457, now=131106-175145.307, channel=TCPChannel [type: server, contact: 127.0.0.1:41205]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:51:45,307-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174944.457, now=131106-175145.307, channel=TCPChannel [type: server, contact: 127.0.0.1:41205]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:51:45,307-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174944.457, now=131106-175145.307, channel=TCPChannel [type: server, contact: 127.0.0.1:41205]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:51:45,307-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174944.457, now=131106-175145.307, channel=TCPChannel [type: server, contact: 127.0.0.1:41205]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:51:45,307-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174944.457, now=131106-175145.307, channel=TCPChannel [type: server, contact: 127.0.0.1:41205]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:51:45,307-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174944.457, now=131106-175145.307, channel=TCPChannel [type: server, contact: 127.0.0.1:41205]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:51:45,307-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174944.457, now=131106-175145.307, channel=TCPChannel [type: server, contact: 127.0.0.1:41205]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:51:45,307-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174944.457, now=131106-175145.307, channel=TCPChannel [type: server, contact: 127.0.0.1:41205]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:51:45,307-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41205] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174944.457, now=131106-175145.307, channel=TCPChannel [type: server, contact: 127.0.0.1:41205]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:51:45,307-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41205] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174944.457, now=131106-175145.307, channel=TCPChannel [type: server, contact: 127.0.0.1:41205]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:51:45,307-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174944.659, now=131106-175145.516, channel=TCPChannel [type: server, contact: 127.0.0.1:41206]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:51:45,516-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174944.659, now=131106-175145.516, channel=TCPChannel [type: server, contact: 127.0.0.1:41206]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:51:45,516-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174944.659, now=131106-175145.516, channel=TCPChannel [type: server, contact: 127.0.0.1:41206]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:51:45,516-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174944.659, now=131106-175145.516, channel=TCPChannel [type: server, contact: 127.0.0.1:41206]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:51:45,516-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174944.659, now=131106-175145.516, channel=TCPChannel [type: server, contact: 127.0.0.1:41206]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:51:45,516-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174944.659, now=131106-175145.516, channel=TCPChannel [type: server, contact: 127.0.0.1:41206]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:51:45,516-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174944.659, now=131106-175145.516, channel=TCPChannel [type: server, contact: 127.0.0.1:41206]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:51:45,516-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174944.659, now=131106-175145.516, channel=TCPChannel [type: server, contact: 127.0.0.1:41206]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:51:45,516-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174944.659, now=131106-175145.516, channel=TCPChannel [type: server, contact: 127.0.0.1:41206]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:51:45,516-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174944.659, now=131106-175145.516, channel=TCPChannel [type: server, contact: 127.0.0.1:41206]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:51:45,516-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41206] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174944.659, now=131106-175145.516, channel=TCPChannel [type: server, contact: 127.0.0.1:41206]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:51:45,516-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41206] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174944.659, now=131106-175145.516, channel=TCPChannel [type: server, contact: 127.0.0.1:41206]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:51:45,516-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174944.861, now=131106-175145.720, channel=TCPChannel [type: server, contact: 127.0.0.1:41207]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:51:45,720-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174944.861, now=131106-175145.720, channel=TCPChannel [type: server, contact: 127.0.0.1:41207]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:51:45,721-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174944.861, now=131106-175145.720, channel=TCPChannel [type: server, contact: 127.0.0.1:41207]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:51:45,721-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174944.861, now=131106-175145.720, channel=TCPChannel [type: server, contact: 127.0.0.1:41207]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:51:45,721-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41207] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174944.861, now=131106-175145.720, channel=TCPChannel [type: server, contact: 127.0.0.1:41207]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:51:45,721-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41207] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-174944.861, now=131106-175145.720, channel=TCPChannel [type: server, contact: 127.0.0.1:41207]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:51:45,721-0600 INFO ChannelManager Channel already shut down
-No streams
-2013-11-06 17:51:53,960-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 17:52:01,581-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 17:52:03,968-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:52:13,976-0600 INFO Multiplexer No streams
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175019.102, now=131106-175219.949, channel=TCPChannel [type: server, contact: 127.0.0.1:41208]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:52:19,949-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175019.102, now=131106-175219.949, channel=TCPChannel [type: server, contact: 127.0.0.1:41208]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:52:19,949-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175019.102, now=131106-175219.949, channel=TCPChannel [type: server, contact: 127.0.0.1:41208]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:52:19,949-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175019.102, now=131106-175219.949, channel=TCPChannel [type: server, contact: 127.0.0.1:41208]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:52:19,949-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175019.102, now=131106-175219.949, channel=TCPChannel [type: server, contact: 127.0.0.1:41208]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:52:19,949-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175019.102, now=131106-175219.949, channel=TCPChannel [type: server, contact: 127.0.0.1:41208]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:52:19,949-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175019.102, now=131106-175219.949, channel=TCPChannel [type: server, contact: 127.0.0.1:41208]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:52:19,949-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175019.102, now=131106-175219.949, channel=TCPChannel [type: server, contact: 127.0.0.1:41208]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:52:19,949-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175019.102, now=131106-175219.949, channel=TCPChannel [type: server, contact: 127.0.0.1:41208]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:52:19,949-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175019.102, now=131106-175219.949, channel=TCPChannel [type: server, contact: 127.0.0.1:41208]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:52:19,949-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41208] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175019.102, now=131106-175219.949, channel=TCPChannel [type: server, contact: 127.0.0.1:41208]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:52:19,949-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41208] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175019.102, now=131106-175219.949, channel=TCPChannel [type: server, contact: 127.0.0.1:41208]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:52:19,949-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175019.304, now=131106-175220.160, channel=TCPChannel [type: server, contact: 127.0.0.1:41209]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:52:20,160-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175019.304, now=131106-175220.160, channel=TCPChannel [type: server, contact: 127.0.0.1:41209]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:52:20,160-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175019.304, now=131106-175220.160, channel=TCPChannel [type: server, contact: 127.0.0.1:41209]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:52:20,160-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175019.304, now=131106-175220.160, channel=TCPChannel [type: server, contact: 127.0.0.1:41209]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:52:20,160-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175019.304, now=131106-175220.160, channel=TCPChannel [type: server, contact: 127.0.0.1:41209]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:52:20,160-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175019.304, now=131106-175220.160, channel=TCPChannel [type: server, contact: 127.0.0.1:41209]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:52:20,160-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175019.304, now=131106-175220.160, channel=TCPChannel [type: server, contact: 127.0.0.1:41209]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:52:20,160-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175019.304, now=131106-175220.160, channel=TCPChannel [type: server, contact: 127.0.0.1:41209]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:52:20,160-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175019.304, now=131106-175220.160, channel=TCPChannel [type: server, contact: 127.0.0.1:41209]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:52:20,160-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175019.304, now=131106-175220.160, channel=TCPChannel [type: server, contact: 127.0.0.1:41209]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:52:20,160-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41209] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175019.304, now=131106-175220.160, channel=TCPChannel [type: server, contact: 127.0.0.1:41209]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:52:20,160-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41209] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175019.304, now=131106-175220.160, channel=TCPChannel [type: server, contact: 127.0.0.1:41209]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:52:20,160-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175019.506, now=131106-175220.364, channel=TCPChannel [type: server, contact: 127.0.0.1:41210]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:52:20,364-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175019.506, now=131106-175220.364, channel=TCPChannel [type: server, contact: 127.0.0.1:41210]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:52:20,365-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175019.506, now=131106-175220.364, channel=TCPChannel [type: server, contact: 127.0.0.1:41210]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:52:20,365-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175019.506, now=131106-175220.364, channel=TCPChannel [type: server, contact: 127.0.0.1:41210]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:52:20,365-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41210] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175019.506, now=131106-175220.364, channel=TCPChannel [type: server, contact: 127.0.0.1:41210]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:52:20,365-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41210] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175019.506, now=131106-175220.364, channel=TCPChannel [type: server, contact: 127.0.0.1:41210]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:52:20,365-0600 INFO ChannelManager Channel already shut down
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:52:22,112-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:52:23,984-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:52:33,991-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:52:38,054-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:52:43,999-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:52:54,007-0600 INFO Multiplexer No streams
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175055.393, now=131106-175256.226, channel=TCPChannel [type: server, contact: 127.0.0.1:41211]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:52:56,226-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175055.393, now=131106-175256.226, channel=TCPChannel [type: server, contact: 127.0.0.1:41211]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:52:56,227-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175055.393, now=131106-175256.226, channel=TCPChannel [type: server, contact: 127.0.0.1:41211]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:52:56,227-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175055.393, now=131106-175256.226, channel=TCPChannel [type: server, contact: 127.0.0.1:41211]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:52:56,227-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175055.393, now=131106-175256.226, channel=TCPChannel [type: server, contact: 127.0.0.1:41211]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:52:56,227-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175055.393, now=131106-175256.226, channel=TCPChannel [type: server, contact: 127.0.0.1:41211]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:52:56,227-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175055.393, now=131106-175256.226, channel=TCPChannel [type: server, contact: 127.0.0.1:41211]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:52:56,227-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175055.393, now=131106-175256.226, channel=TCPChannel [type: server, contact: 127.0.0.1:41211]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:52:56,227-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175055.393, now=131106-175256.226, channel=TCPChannel [type: server, contact: 127.0.0.1:41211]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:52:56,227-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175055.393, now=131106-175256.226, channel=TCPChannel [type: server, contact: 127.0.0.1:41211]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:52:56,227-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41211] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175055.393, now=131106-175256.226, channel=TCPChannel [type: server, contact: 127.0.0.1:41211]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:52:56,227-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41211] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175055.393, now=131106-175256.226, channel=TCPChannel [type: server, contact: 127.0.0.1:41211]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:52:56,227-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175055.595, now=131106-175256.439, channel=TCPChannel [type: server, contact: 127.0.0.1:41212]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:52:56,439-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175055.595, now=131106-175256.439, channel=TCPChannel [type: server, contact: 127.0.0.1:41212]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:52:56,439-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175055.595, now=131106-175256.439, channel=TCPChannel [type: server, contact: 127.0.0.1:41212]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:52:56,439-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175055.595, now=131106-175256.439, channel=TCPChannel [type: server, contact: 127.0.0.1:41212]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 17:52:56,439-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175055.595, now=131106-175256.439, channel=TCPChannel [type: server, contact: 127.0.0.1:41212]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:52:56,439-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175055.595, now=131106-175256.439, channel=TCPChannel [type: server, contact: 127.0.0.1:41212]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 17:52:56,440-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175055.595, now=131106-175256.439, channel=TCPChannel [type: server, contact: 127.0.0.1:41212]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:52:56,440-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175055.595, now=131106-175256.439, channel=TCPChannel [type: server, contact: 127.0.0.1:41212]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 17:52:56,440-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175055.595, now=131106-175256.439, channel=TCPChannel [type: server, contact: 127.0.0.1:41212]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:52:56,440-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175055.595, now=131106-175256.439, channel=TCPChannel [type: server, contact: 127.0.0.1:41212]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:52:56,440-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41212] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175055.595, now=131106-175256.439, channel=TCPChannel [type: server, contact: 127.0.0.1:41212]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:52:56,440-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41212] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175055.595, now=131106-175256.439, channel=TCPChannel [type: server, contact: 127.0.0.1:41212]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:52:56,440-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175055.796, now=131106-175256.652, channel=TCPChannel [type: server, contact: 127.0.0.1:41213]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:52:56,652-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175055.796, now=131106-175256.652, channel=TCPChannel [type: server, contact: 127.0.0.1:41213]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 17:52:56,652-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175055.796, now=131106-175256.652, channel=TCPChannel [type: server, contact: 127.0.0.1:41213]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:52:56,652-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175055.796, now=131106-175256.652, channel=TCPChannel [type: server, contact: 127.0.0.1:41213]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 17:52:56,652-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41213] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175055.796, now=131106-175256.652, channel=TCPChannel [type: server, contact: 127.0.0.1:41213]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 17:52:56,652-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41213] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-175055.796, now=131106-175256.652, channel=TCPChannel [type: server, contact: 127.0.0.1:41213]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 17:52:56,652-0600 INFO ChannelManager Channel already shut down
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-06 17:52:59,430-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-06 17:53:04,014-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:53:14,022-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:53:14,528-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:53:24,030-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:53:34,037-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:53:36,748-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:53:44,045-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 15ms
-2013-11-06 17:53:51,016-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 15ms
-No streams
-2013-11-06 17:53:54,053-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:54:04,060-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:54:14,066-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:54:14,068-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:54:24,076-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:54:27,476-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:54:34,083-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:54:44,091-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-06 17:54:51,385-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-06 17:54:54,099-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 17:55:03,951-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 17:55:04,107-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:55:14,114-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:55:24,122-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:55:28,703-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:55:34,130-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:55:40,424-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:55:44,137-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:55:54,145-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:56:04,153-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:56:06,021-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:56:14,161-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:56:16,898-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:56:24,168-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:56:34,176-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-06 17:56:43,340-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-06 17:56:44,184-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:56:53,372-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:56:54,191-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:57:04,199-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:57:14,206-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:57:20,658-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:57:24,214-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:57:29,846-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:57:34,221-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:57:44,229-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:57:54,237-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-06 17:57:57,976-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-06 17:58:04,244-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 4ms
-2013-11-06 17:58:06,323-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 4ms
-No streams
-2013-11-06 17:58:14,251-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:58:24,259-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:58:34,267-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:58:35,294-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:58:42,794-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:58:44,274-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:58:54,282-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:59:04,290-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:59:12,613-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:59:14,298-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 17:59:19,269-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 17:59:24,306-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:59:34,314-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 17:59:44,322-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:59:49,930-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 17:59:54,330-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 17:59:55,742-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 18:00:04,339-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:00:14,347-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:00:24,355-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:00:27,249-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:00:32,217-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 18:00:34,363-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:00:44,371-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:00:54,381-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:01:04,389-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-06 18:01:04,568-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:01:08,690-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 18:01:14,398-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:01:24,409-0600 INFO Multiplexer No streams
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 18:01:30,507-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41223] for addition
-2013-11-06 18:01:30,507-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41223] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41223]
-2013-11-06 18:01:30,507-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41223]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f81
-2013-11-06 18:01:30,529-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f81
-MetaChannel [context: cps-41223, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41223]
-2013-11-06 18:01:30,529-0600 INFO MetaChannel MetaChannel [context: cps-41223, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41223]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 18:01:30,529-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 18:01:30,570-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 18:01:30,570-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 18:01:30,650-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 18:01:30,658-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 87.0) submitting urn:job-0-1383759274828-1383759274829: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 18:01:30,658-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 87.0) submitting urn:job-0-1383759274828-1383759274829: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 18:01:30,658-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 18:01:30,658-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-06 18:01:30,664-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7506, JOBSTATUS) unregistering (send)
-2013-11-06 18:01:30,665-0600 INFO RequestHandler Handler(tag: 7506, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 18:01:30,708-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41224] for addition
-2013-11-06 18:01:30,709-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41224] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41224]
-2013-11-06 18:01:30,709-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41224]
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f80
-2013-11-06 18:01:30,731-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f80
-MetaChannel [context: cps-41224, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41224]
-2013-11-06 18:01:30,731-0600 INFO MetaChannel MetaChannel [context: cps-41224, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41224]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-06 18:01:30,731-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-06 18:01:30,772-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 18:01:30,873-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 18:01:30,909-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-crush.mcs.anl.gov:0 (quality: 28.0) submitting urn:job-1-1383759274831-1383759274832: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 18:01:30,909-0600 INFO Cpu crush.mcs.anl.gov:0 (quality: 28.0) submitting urn:job-1-1383759274831-1383759274832: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 18:01:30,916-0600 DEBUG Cpu Active workerid=crush.mcs.anl.gov:000000
-Handler(tag: 87, JOBSTATUS) unregistering (send)
-2013-11-06 18:01:30,917-0600 INFO RequestHandler Handler(tag: 87, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 18:01:30,920-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41225] for addition
-2013-11-06 18:01:30,920-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41225] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41225]
-2013-11-06 18:01:30,920-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41225]
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f7f
-2013-11-06 18:01:30,954-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f7f
-MetaChannel [context: cps-41225, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41225]
-2013-11-06 18:01:30,954-0600 INFO MetaChannel MetaChannel [context: cps-41225, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41225]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-06 18:01:30,954-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-06 18:01:31,014-0600 INFO RequestHandler Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-06 18:01:31,095-0600 INFO RequestHandler Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-06 18:01:32,683-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 18:01:32,683-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 18:01:32,683-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-06 18:01:32,683-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-Handler(tag: 7507, JOBSTATUS) unregistering (send)
-2013-11-06 18:01:32,683-0600 INFO RequestHandler Handler(tag: 7507, JOBSTATUS) unregistering (send)
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 18:01:32,683-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-grind.mcs.anl.gov:0 (quality: 88.0) submitting urn:job-2-1383759274834-1383759274835: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 18:01:32,684-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 88.0) submitting urn:job-2-1383759274834-1383759274835: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-Channel config: null -> {keepalive=null}
-2013-11-06 18:01:32,684-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41223] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 18:01:32,684-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41223] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-06 18:01:32,685-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-41223, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41223]] binding to NullChannel
-2013-11-06 18:01:32,685-0600 INFO MetaChannel MetaChannel [context: cps-41223, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41223]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41223]
-2013-11-06 18:01:32,685-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41223]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 18:01:32,685-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 18:01:32,685-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 18:01:32,685-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 18:01:32,685-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 18:01:32,685-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 18:01:32,685-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 18:01:32,685-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 18:01:32,685-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 18:01:32,685-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 18:01:32,686-0600 INFO ChannelManager Channel exception handled
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 18:01:32,686-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel config: null -> {keepalive=null}
-2013-11-06 18:01:32,686-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41225] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 18:01:32,686-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41225] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-06 18:01:32,686-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-41225, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41225]] binding to NullChannel
-2013-11-06 18:01:32,686-0600 INFO MetaChannel MetaChannel [context: cps-41225, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41225]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41225]
-2013-11-06 18:01:32,686-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41225]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 18:01:32,686-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 18:01:32,686-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 18:01:32,686-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 18:01:32,686-0600 INFO ChannelManager Channel exception handled
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 18:01:32,689-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 18:01:32,690-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 18:01:32,690-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7508, JOBSTATUS) unregistering (send)
-2013-11-06 18:01:32,690-0600 INFO RequestHandler Handler(tag: 7508, JOBSTATUS) unregistering (send)
-Channel is closed. Removing.
-2013-11-06 18:01:32,700-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 18:01:32,700-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 18:01:32,701-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41225] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 18:01:32,701-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41225] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 18:01:32,701-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41225]
-2013-11-06 18:01:32,701-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41225]
-Channel is closed. Removing.
-2013-11-06 18:01:32,701-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 18:01:32,701-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 18:01:32,701-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41223] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 18:01:32,701-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41223] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 18:01:32,701-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41223]
-2013-11-06 18:01:32,701-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41223]
-2013-11-06 18:01:33,296-0600 DEBUG Cpu Completed
-crush.mcs.anl.gov:0 jobTerminated
-2013-11-06 18:01:33,296-0600 INFO Cpu crush.mcs.anl.gov:0 jobTerminated
-2013-11-06 18:01:33,296-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 18:01:33,296-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 18:01:33,296-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-Channel config: null -> {keepalive=null}
-2013-11-06 18:01:33,296-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-06 18:01:33,297-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-TCPChannel [type: server, contact: 127.0.0.1:41224] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 18:01:33,297-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41224] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 88, JOBSTATUS) unregistering (send)
-2013-11-06 18:01:33,297-0600 INFO RequestHandler Handler(tag: 88, JOBSTATUS) unregistering (send)
-null: Channel shut down
-2013-11-06 18:01:33,297-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-41224, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41224]] binding to NullChannel
-2013-11-06 18:01:33,297-0600 INFO MetaChannel MetaChannel [context: cps-41224, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41224]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41224]
-2013-11-06 18:01:33,297-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41224]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 18:01:33,297-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 18:01:33,297-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 18:01:33,297-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 18:01:33,298-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 18:01:33,298-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 18:01:33,298-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 18:01:33,298-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 18:01:33,298-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 18:01:33,298-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 18:01:33,298-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 18:01:33,303-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 18:01:33,303-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 18:01:33,303-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41224] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 18:01:33,303-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41224] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 18:01:33,303-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41224]
-2013-11-06 18:01:33,303-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41224]
-No streams
-2013-11-06 18:01:34,426-0600 INFO Multiplexer No streams
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 18:01:34,703-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 18:01:34,703-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 18:01:34,704-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 18:01:34,704-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-06 18:01:34,704-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-2013-11-06 18:01:34,704-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-Handler(tag: 7509, JOBSTATUS) unregistering (send)
-2013-11-06 18:01:34,704-0600 INFO RequestHandler Handler(tag: 7509, JOBSTATUS) unregistering (send)
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:01:41,886-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 18:01:44,437-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:01:45,164-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 18:01:54,447-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:02:04,456-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:02:14,465-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-06 18:02:19,205-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 18:02:20,768-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41226] for addition
-2013-11-06 18:02:20,768-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41226] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41226]
-2013-11-06 18:02:20,768-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41226]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f7e
-2013-11-06 18:02:20,805-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f7e
-MetaChannel [context: cps-41226, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41226]
-2013-11-06 18:02:20,805-0600 INFO MetaChannel MetaChannel [context: cps-41226, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41226]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 18:02:20,805-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 18:02:20,846-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 18:02:20,846-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 18:02:20,926-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 18:02:20,967-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 89.0) submitting urn:job-0-1383759274837-1383759274838: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 18:02:20,967-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 89.0) submitting urn:job-0-1383759274837-1383759274838: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 18:02:20,967-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 18:02:20,967-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-06 18:02:20,972-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7510, JOBSTATUS) unregistering (send)
-2013-11-06 18:02:20,973-0600 INFO RequestHandler Handler(tag: 7510, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 18:02:20,984-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41227] for addition
-2013-11-06 18:02:20,984-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41227] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41227]
-2013-11-06 18:02:20,984-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41227]
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f7d
-2013-11-06 18:02:21,007-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f7d
-MetaChannel [context: cps-41227, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41227]
-2013-11-06 18:02:21,007-0600 INFO MetaChannel MetaChannel [context: cps-41227, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41227]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-06 18:02:21,007-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-06 18:02:21,047-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 18:02:21,128-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 18:02:21,168-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-crush.mcs.anl.gov:0 (quality: 29.0) submitting urn:job-1-1383759274840-1383759274841: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 18:02:21,168-0600 INFO Cpu crush.mcs.anl.gov:0 (quality: 29.0) submitting urn:job-1-1383759274840-1383759274841: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 18:02:21,188-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41228] for addition
-2013-11-06 18:02:21,188-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41228] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41228]
-2013-11-06 18:02:21,188-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41228]
-2013-11-06 18:02:21,199-0600 DEBUG Cpu Active workerid=crush.mcs.anl.gov:000000
-Handler(tag: 89, JOBSTATUS) unregistering (send)
-2013-11-06 18:02:21,199-0600 INFO RequestHandler Handler(tag: 89, JOBSTATUS) unregistering (send)
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f7c
-2013-11-06 18:02:21,210-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f7c
-MetaChannel [context: cps-41228, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41228]
-2013-11-06 18:02:21,210-0600 INFO MetaChannel MetaChannel [context: cps-41228, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41228]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-06 18:02:21,210-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-06 18:02:21,251-0600 INFO RequestHandler Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-06 18:02:21,332-0600 INFO RequestHandler Handler(tag: 42566, SUBMITJOB) unregistering (send)
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:02:21,638-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:02:22,985-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 18:02:22,985-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 18:02:22,985-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 18:02:22,985-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 7511, JOBSTATUS) unregistering (send)
-2013-11-06 18:02:22,985-0600 INFO RequestHandler Handler(tag: 7511, JOBSTATUS) unregistering (send)
-Channel config: null -> {keepalive=null}
-2013-11-06 18:02:22,986-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41226] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 18:02:22,986-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41226] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 18:02:22,986-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-null: Channel shut down
-2013-11-06 18:02:22,986-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-41226, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41226]] binding to NullChannel
-2013-11-06 18:02:22,986-0600 INFO MetaChannel MetaChannel [context: cps-41226, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41226]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41226]
-2013-11-06 18:02:22,986-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41226]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 18:02:22,987-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 18:02:22,988-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 18:02:22,988-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-grind.mcs.anl.gov:0 (quality: 90.0) submitting urn:job-2-1383759274843-1383759274844: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 18:02:22,988-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 90.0) submitting urn:job-2-1383759274843-1383759274844: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 18:02:22,988-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 18:02:22,988-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 18:02:22,988-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 18:02:22,988-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 18:02:22,991-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 18:02:22,991-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 18:02:22,995-0600 INFO ChannelManager Channel exception handled
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 18:02:22,995-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel config: null -> {keepalive=null}
-2013-11-06 18:02:22,995-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41228] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 18:02:22,995-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41228] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-06 18:02:22,995-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-41228, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41228]] binding to NullChannel
-2013-11-06 18:02:22,995-0600 INFO MetaChannel MetaChannel [context: cps-41228, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41228]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41228]
-2013-11-06 18:02:22,995-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41228]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 18:02:22,995-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 18:02:22,996-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 18:02:22,996-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 18:02:22,996-0600 INFO ChannelManager Channel exception handled
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 18:02:22,996-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-Channel is closed. Removing.
-2013-11-06 18:02:22,997-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 18:02:22,998-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 18:02:22,998-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41226] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 18:02:22,998-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41226] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 18:02:22,998-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41226]
-2013-11-06 18:02:22,998-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41226]
-Channel is closed. Removing.
-2013-11-06 18:02:22,998-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 18:02:22,998-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 18:02:22,998-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 18:02:22,998-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 18:02:22,998-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-TCPChannel [type: server, contact: 127.0.0.1:41228] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 18:02:22,998-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41228] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 18:02:22,998-0600 INFO ChannelManager Channel already shut down
-Handler(tag: 7512, JOBSTATUS) unregistering (send)
-2013-11-06 18:02:22,998-0600 INFO RequestHandler Handler(tag: 7512, JOBSTATUS) unregistering (send)
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41228]
-2013-11-06 18:02:22,999-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41228]
-2013-11-06 18:02:23,213-0600 DEBUG Cpu Completed
-crush.mcs.anl.gov:0 jobTerminated
-2013-11-06 18:02:23,213-0600 INFO Cpu crush.mcs.anl.gov:0 jobTerminated
-2013-11-06 18:02:23,213-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 18:02:23,213-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 90, JOBSTATUS) unregistering (send)
-2013-11-06 18:02:23,213-0600 INFO RequestHandler Handler(tag: 90, JOBSTATUS) unregistering (send)
-2013-11-06 18:02:23,213-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-Channel config: null -> {keepalive=null}
-2013-11-06 18:02:23,213-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-06 18:02:23,214-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-TCPChannel [type: server, contact: 127.0.0.1:41227] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 18:02:23,214-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41227] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-06 18:02:23,214-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-41227, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41227]] binding to NullChannel
-2013-11-06 18:02:23,214-0600 INFO MetaChannel MetaChannel [context: cps-41227, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41227]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41227]
-2013-11-06 18:02:23,214-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41227]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 18:02:23,214-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 18:02:23,214-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 18:02:23,214-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 18:02:23,214-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 18:02:23,214-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 18:02:23,215-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 18:02:23,215-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 18:02:23,215-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 18:02:23,215-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 18:02:23,215-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 18:02:23,219-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 18:02:23,220-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 18:02:23,220-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41227] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 18:02:23,220-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41227] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 18:02:23,220-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41227]
-2013-11-06 18:02:23,220-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41227]
-No streams
-2013-11-06 18:02:24,484-0600 INFO Multiplexer No streams
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 18:02:25,009-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 18:02:25,009-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 18:02:25,009-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 18:02:25,009-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-06 18:02:25,009-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7513, JOBSTATUS) unregistering (send)
-2013-11-06 18:02:25,009-0600 INFO RequestHandler Handler(tag: 7513, JOBSTATUS) unregistering (send)
-2013-11-06 18:02:25,010-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-No streams
-2013-11-06 18:02:34,494-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:02:44,503-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:02:54,516-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-06 18:02:56,523-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:02:58,112-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 18:03:04,527-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:03:14,539-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:03:24,550-0600 INFO Multiplexer No streams
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180130.650, now=131106-180331.507, channel=TCPChannel [type: server, contact: 127.0.0.1:41223]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 18:03:31,507-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180130.650, now=131106-180331.507, channel=TCPChannel [type: server, contact: 127.0.0.1:41223]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 18:03:31,507-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180130.650, now=131106-180331.507, channel=TCPChannel [type: server, contact: 127.0.0.1:41223]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 18:03:31,507-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180130.650, now=131106-180331.507, channel=TCPChannel [type: server, contact: 127.0.0.1:41223]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 18:03:31,507-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180130.650, now=131106-180331.507, channel=TCPChannel [type: server, contact: 127.0.0.1:41223]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 18:03:31,508-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180130.650, now=131106-180331.507, channel=TCPChannel [type: server, contact: 127.0.0.1:41223]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 18:03:31,508-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180130.650, now=131106-180331.507, channel=TCPChannel [type: server, contact: 127.0.0.1:41223]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 18:03:31,508-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180130.650, now=131106-180331.507, channel=TCPChannel [type: server, contact: 127.0.0.1:41223]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 18:03:31,508-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180130.650, now=131106-180331.507, channel=TCPChannel [type: server, contact: 127.0.0.1:41223]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 18:03:31,508-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180130.650, now=131106-180331.507, channel=TCPChannel [type: server, contact: 127.0.0.1:41223]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 18:03:31,508-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41223] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180130.650, now=131106-180331.507, channel=TCPChannel [type: server, contact: 127.0.0.1:41223]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 18:03:31,508-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41223] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180130.650, now=131106-180331.507, channel=TCPChannel [type: server, contact: 127.0.0.1:41223]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 18:03:31,508-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180130.873, now=131106-180331.708, channel=TCPChannel [type: server, contact: 127.0.0.1:41224]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 18:03:31,708-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180130.873, now=131106-180331.708, channel=TCPChannel [type: server, contact: 127.0.0.1:41224]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 18:03:31,708-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180130.873, now=131106-180331.708, channel=TCPChannel [type: server, contact: 127.0.0.1:41224]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 18:03:31,709-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180130.873, now=131106-180331.708, channel=TCPChannel [type: server, contact: 127.0.0.1:41224]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 18:03:31,709-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180130.873, now=131106-180331.708, channel=TCPChannel [type: server, contact: 127.0.0.1:41224]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 18:03:31,709-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180130.873, now=131106-180331.708, channel=TCPChannel [type: server, contact: 127.0.0.1:41224]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 18:03:31,709-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180130.873, now=131106-180331.708, channel=TCPChannel [type: server, contact: 127.0.0.1:41224]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 18:03:31,709-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180130.873, now=131106-180331.708, channel=TCPChannel [type: server, contact: 127.0.0.1:41224]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 18:03:31,709-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180130.873, now=131106-180331.708, channel=TCPChannel [type: server, contact: 127.0.0.1:41224]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 18:03:31,709-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180130.873, now=131106-180331.708, channel=TCPChannel [type: server, contact: 127.0.0.1:41224]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 18:03:31,709-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41224] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180130.873, now=131106-180331.708, channel=TCPChannel [type: server, contact: 127.0.0.1:41224]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 18:03:31,709-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41224] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180130.873, now=131106-180331.708, channel=TCPChannel [type: server, contact: 127.0.0.1:41224]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 18:03:31,709-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180131.095, now=131106-180331.920, channel=TCPChannel [type: server, contact: 127.0.0.1:41225]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 18:03:31,921-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180131.095, now=131106-180331.920, channel=TCPChannel [type: server, contact: 127.0.0.1:41225]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 18:03:31,921-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180131.095, now=131106-180331.920, channel=TCPChannel [type: server, contact: 127.0.0.1:41225]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 18:03:31,921-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180131.095, now=131106-180331.920, channel=TCPChannel [type: server, contact: 127.0.0.1:41225]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 18:03:31,921-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41225] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180131.095, now=131106-180331.920, channel=TCPChannel [type: server, contact: 127.0.0.1:41225]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 18:03:31,921-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41225] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180131.095, now=131106-180331.920, channel=TCPChannel [type: server, contact: 127.0.0.1:41225]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 18:03:31,921-0600 INFO ChannelManager Channel already shut down
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-06 18:03:33,841-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-No streams
-2013-11-06 18:03:34,568-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:03:34,586-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 18:03:44,584-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:03:54,603-0600 INFO Multiplexer No streams
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 18:04:04,155-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41234] for addition
-2013-11-06 18:04:04,155-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41234] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41234]
-2013-11-06 18:04:04,155-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41234]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f7b
-2013-11-06 18:04:04,191-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f7b
-MetaChannel [context: cps-41234, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41234]
-2013-11-06 18:04:04,191-0600 INFO MetaChannel MetaChannel [context: cps-41234, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41234]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-06 18:04:04,191-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-06 18:04:04,232-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-06 18:04:04,232-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 18:04:04,312-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-06 18:04:04,314-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 91.0) submitting urn:job-0-1383759274846-1383759274847: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 18:04:04,314-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 91.0) submitting urn:job-0-1383759274846-1383759274847: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 18:04:04,316-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 18:04:04,316-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-06 18:04:04,321-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7514, JOBSTATUS) unregistering (send)
-2013-11-06 18:04:04,322-0600 INFO RequestHandler Handler(tag: 7514, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 18:04:04,372-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41235] for addition
-2013-11-06 18:04:04,372-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41235] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41235]
-2013-11-06 18:04:04,372-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41235]
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f7a
-2013-11-06 18:04:04,393-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f7a
-MetaChannel [context: cps-41235, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41235]
-2013-11-06 18:04:04,393-0600 INFO MetaChannel MetaChannel [context: cps-41235, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41235]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-06 18:04:04,393-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-06 18:04:04,433-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 18:04:04,514-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-06 18:04:04,516-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-crush.mcs.anl.gov:0 (quality: 30.0) submitting urn:job-1-1383759274849-1383759274850: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 18:04:04,516-0600 INFO Cpu crush.mcs.anl.gov:0 (quality: 30.0) submitting urn:job-1-1383759274849-1383759274850: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 18:04:04,522-0600 DEBUG Cpu Active workerid=crush.mcs.anl.gov:000000
-Handler(tag: 91, JOBSTATUS) unregistering (send)
-2013-11-06 18:04:04,522-0600 INFO RequestHandler Handler(tag: 91, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-06 18:04:04,573-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41236] for addition
-2013-11-06 18:04:04,573-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:41236] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:41236]
-2013-11-06 18:04:04,573-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:41236]
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f79
-2013-11-06 18:04:04,594-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f79
-MetaChannel [context: cps-41236, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41236]
-2013-11-06 18:04:04,594-0600 INFO MetaChannel MetaChannel [context: cps-41236, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:41236]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-06 18:04:04,595-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-Avg stream buf: 6
-2013-11-06 18:04:04,615-0600 INFO Multiplexer Avg stream buf: 6
-Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-06 18:04:04,635-0600 INFO RequestHandler Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-06 18:04:04,715-0600 INFO RequestHandler Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-06 18:04:06,338-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 18:04:06,339-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 18:04:06,339-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 18:04:06,339-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 7515, JOBSTATUS) unregistering (send)
-2013-11-06 18:04:06,339-0600 INFO RequestHandler Handler(tag: 7515, JOBSTATUS) unregistering (send)
-2013-11-06 18:04:06,339-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-Channel config: null -> {keepalive=null}
-2013-11-06 18:04:06,339-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41234] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 18:04:06,339-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41234] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-grind.mcs.anl.gov:0 (quality: 92.0) submitting urn:job-2-1383759274852-1383759274853: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-06 18:04:06,339-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 92.0) submitting urn:job-2-1383759274852-1383759274853: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-null: Channel shut down
-2013-11-06 18:04:06,340-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-41234, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41234]] binding to NullChannel
-2013-11-06 18:04:06,340-0600 INFO MetaChannel MetaChannel [context: cps-41234, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41234]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41234]
-2013-11-06 18:04:06,340-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41234]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 18:04:06,340-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 18:04:06,340-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 18:04:06,340-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 18:04:06,340-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 18:04:06,340-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 18:04:06,341-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 18:04:06,341-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 18:04:06,341-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 18:04:06,341-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 18:04:06,341-0600 INFO ChannelManager Channel exception handled
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 18:04:06,341-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel config: null -> {keepalive=null}
-2013-11-06 18:04:06,341-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41236] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 18:04:06,341-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41236] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel is closed. Removing.
-2013-11-06 18:04:06,341-0600 INFO Multiplexer Channel is closed. Removing.
-null: Channel shut down
-2013-11-06 18:04:06,342-0600 INFO AbstractTCPChannel null: Channel shut down
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 18:04:06,342-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-MetaChannel [context: cps-41236, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41236]] binding to NullChannel
-2013-11-06 18:04:06,342-0600 INFO MetaChannel MetaChannel [context: cps-41236, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41236]] binding to NullChannel
-Channel config: null -> {keepalive=null}
-2013-11-06 18:04:06,342-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41236]
-2013-11-06 18:04:06,342-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41236]
-TCPChannel [type: server, contact: 127.0.0.1:41234] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 18:04:06,342-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41234] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 18:04:06,342-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 18:04:06,342-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41234]
-2013-11-06 18:04:06,342-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41234]
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 18:04:06,343-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 18:04:06,343-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel is closed. Removing.
-2013-11-06 18:04:06,343-0600 INFO Multiplexer Channel is closed. Removing.
-Channel exception handled
-2013-11-06 18:04:06,343-0600 INFO ChannelManager Channel exception handled
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 18:04:06,343-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 18:04:06,343-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41236] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 18:04:06,343-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41236] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 18:04:06,343-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41236]
-2013-11-06 18:04:06,343-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41236]
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 18:04:06,375-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 18:04:06,376-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 18:04:06,376-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7516, JOBSTATUS) unregistering (send)
-2013-11-06 18:04:06,376-0600 INFO RequestHandler Handler(tag: 7516, JOBSTATUS) unregistering (send)
-2013-11-06 18:04:06,540-0600 DEBUG Cpu Completed
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 18:04:06,540-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-crush.mcs.anl.gov:0 jobTerminated
-2013-11-06 18:04:06,541-0600 INFO Cpu crush.mcs.anl.gov:0 jobTerminated
-Channel config: null -> {keepalive=null}
-2013-11-06 18:04:06,541-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-06 18:04:06,541-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-TCPChannel [type: server, contact: 127.0.0.1:41235] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 18:04:06,541-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41235] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 92, JOBSTATUS) unregistering (send)
-2013-11-06 18:04:06,541-0600 INFO RequestHandler Handler(tag: 92, JOBSTATUS) unregistering (send)
-2013-11-06 18:04:06,541-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-06 18:04:06,542-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-null: Channel shut down
-2013-11-06 18:04:06,542-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-41235, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41235]] binding to NullChannel
-2013-11-06 18:04:06,542-0600 INFO MetaChannel MetaChannel [context: cps-41235, boundTo: TCPChannel [type: server, contact: 127.0.0.1:41235]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41235]
-2013-11-06 18:04:06,542-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:41235]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 18:04:06,542-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 18:04:06,542-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 18:04:06,542-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 18:04:06,542-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 18:04:06,542-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 18:04:06,543-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 18:04:06,543-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 18:04:06,543-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-06 18:04:06,543-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-06 18:04:06,543-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-06 18:04:06,544-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 18:04:06,544-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-06 18:04:06,544-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41235] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-06 18:04:06,544-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41235] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-06 18:04:06,544-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41235]
-2013-11-06 18:04:06,544-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:41235]
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 18:04:08,392-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-06 18:04:08,392-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 18:04:08,392-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-06 18:04:08,392-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-06 18:04:08,392-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7517, JOBSTATUS) unregistering (send)
-2013-11-06 18:04:08,392-0600 INFO RequestHandler Handler(tag: 7517, JOBSTATUS) unregistering (send)
-2013-11-06 18:04:08,392-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 18:04:11,061-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-06 18:04:11,159-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-No streams
-2013-11-06 18:04:14,631-0600 INFO Multiplexer No streams
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180220.926, now=131106-180421.768, channel=TCPChannel [type: server, contact: 127.0.0.1:41226]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 18:04:21,769-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180220.926, now=131106-180421.768, channel=TCPChannel [type: server, contact: 127.0.0.1:41226]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 18:04:21,769-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180220.926, now=131106-180421.768, channel=TCPChannel [type: server, contact: 127.0.0.1:41226]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 18:04:21,769-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180220.926, now=131106-180421.768, channel=TCPChannel [type: server, contact: 127.0.0.1:41226]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 18:04:21,769-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180220.926, now=131106-180421.768, channel=TCPChannel [type: server, contact: 127.0.0.1:41226]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 18:04:21,769-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180220.926, now=131106-180421.768, channel=TCPChannel [type: server, contact: 127.0.0.1:41226]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 18:04:21,769-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180220.926, now=131106-180421.768, channel=TCPChannel [type: server, contact: 127.0.0.1:41226]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 18:04:21,769-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180220.926, now=131106-180421.768, channel=TCPChannel [type: server, contact: 127.0.0.1:41226]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 18:04:21,769-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180220.926, now=131106-180421.768, channel=TCPChannel [type: server, contact: 127.0.0.1:41226]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 18:04:21,769-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180220.926, now=131106-180421.768, channel=TCPChannel [type: server, contact: 127.0.0.1:41226]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 18:04:21,769-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41226] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180220.926, now=131106-180421.768, channel=TCPChannel [type: server, contact: 127.0.0.1:41226]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 18:04:21,769-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41226] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180220.926, now=131106-180421.768, channel=TCPChannel [type: server, contact: 127.0.0.1:41226]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 18:04:21,769-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180221.128, now=131106-180421.984, channel=TCPChannel [type: server, contact: 127.0.0.1:41227]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 18:04:21,984-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180221.128, now=131106-180421.984, channel=TCPChannel [type: server, contact: 127.0.0.1:41227]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 18:04:21,984-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180221.128, now=131106-180421.984, channel=TCPChannel [type: server, contact: 127.0.0.1:41227]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 18:04:21,984-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180221.128, now=131106-180421.984, channel=TCPChannel [type: server, contact: 127.0.0.1:41227]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 18:04:21,984-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180221.128, now=131106-180421.984, channel=TCPChannel [type: server, contact: 127.0.0.1:41227]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 18:04:21,984-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180221.128, now=131106-180421.984, channel=TCPChannel [type: server, contact: 127.0.0.1:41227]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 18:04:21,984-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180221.128, now=131106-180421.984, channel=TCPChannel [type: server, contact: 127.0.0.1:41227]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 18:04:21,984-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180221.128, now=131106-180421.984, channel=TCPChannel [type: server, contact: 127.0.0.1:41227]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 18:04:21,984-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180221.128, now=131106-180421.984, channel=TCPChannel [type: server, contact: 127.0.0.1:41227]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 18:04:21,985-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180221.128, now=131106-180421.984, channel=TCPChannel [type: server, contact: 127.0.0.1:41227]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 18:04:21,985-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41227] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180221.128, now=131106-180421.984, channel=TCPChannel [type: server, contact: 127.0.0.1:41227]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 18:04:21,985-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41227] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180221.128, now=131106-180421.984, channel=TCPChannel [type: server, contact: 127.0.0.1:41227]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 18:04:21,985-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180221.332, now=131106-180422.188, channel=TCPChannel [type: server, contact: 127.0.0.1:41228]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 18:04:22,188-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180221.332, now=131106-180422.188, channel=TCPChannel [type: server, contact: 127.0.0.1:41228]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 18:04:22,188-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180221.332, now=131106-180422.188, channel=TCPChannel [type: server, contact: 127.0.0.1:41228]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 18:04:22,189-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180221.332, now=131106-180422.188, channel=TCPChannel [type: server, contact: 127.0.0.1:41228]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 18:04:22,189-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41228] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180221.332, now=131106-180422.188, channel=TCPChannel [type: server, contact: 127.0.0.1:41228]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 18:04:22,189-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41228] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180221.332, now=131106-180422.188, channel=TCPChannel [type: server, contact: 127.0.0.1:41228]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 18:04:22,189-0600 INFO ChannelManager Channel already shut down
-No streams
-2013-11-06 18:04:24,646-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:04:34,660-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:04:44,669-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 18:04:47,536-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:04:48,476-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 18:04:54,677-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:05:04,686-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:05:14,698-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:05:24,010-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 18:05:24,709-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:05:25,795-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 18:05:34,720-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:05:44,729-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:05:54,739-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:06:00,483-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:06:03,113-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 18:06:04,749-0600 INFO Multiplexer No streams
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180404.312, now=131106-180605.155, channel=TCPChannel [type: server, contact: 127.0.0.1:41234]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 18:06:05,155-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180404.312, now=131106-180605.155, channel=TCPChannel [type: server, contact: 127.0.0.1:41234]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 18:06:05,155-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180404.312, now=131106-180605.155, channel=TCPChannel [type: server, contact: 127.0.0.1:41234]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 18:06:05,155-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180404.312, now=131106-180605.155, channel=TCPChannel [type: server, contact: 127.0.0.1:41234]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 18:06:05,155-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180404.312, now=131106-180605.155, channel=TCPChannel [type: server, contact: 127.0.0.1:41234]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 18:06:05,155-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180404.312, now=131106-180605.155, channel=TCPChannel [type: server, contact: 127.0.0.1:41234]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 18:06:05,156-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180404.312, now=131106-180605.155, channel=TCPChannel [type: server, contact: 127.0.0.1:41234]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 18:06:05,156-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180404.312, now=131106-180605.155, channel=TCPChannel [type: server, contact: 127.0.0.1:41234]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 18:06:05,156-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180404.312, now=131106-180605.155, channel=TCPChannel [type: server, contact: 127.0.0.1:41234]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 18:06:05,156-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180404.312, now=131106-180605.155, channel=TCPChannel [type: server, contact: 127.0.0.1:41234]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 18:06:05,156-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41234] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180404.312, now=131106-180605.155, channel=TCPChannel [type: server, contact: 127.0.0.1:41234]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 18:06:05,156-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41234] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180404.312, now=131106-180605.155, channel=TCPChannel [type: server, contact: 127.0.0.1:41234]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 18:06:05,156-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180404.514, now=131106-180605.372, channel=TCPChannel [type: server, contact: 127.0.0.1:41235]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 18:06:05,372-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180404.514, now=131106-180605.372, channel=TCPChannel [type: server, contact: 127.0.0.1:41235]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 18:06:05,372-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180404.514, now=131106-180605.372, channel=TCPChannel [type: server, contact: 127.0.0.1:41235]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 18:06:05,372-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180404.514, now=131106-180605.372, channel=TCPChannel [type: server, contact: 127.0.0.1:41235]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-06 18:06:05,372-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180404.514, now=131106-180605.372, channel=TCPChannel [type: server, contact: 127.0.0.1:41235]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 18:06:05,372-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180404.514, now=131106-180605.372, channel=TCPChannel [type: server, contact: 127.0.0.1:41235]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-06 18:06:05,373-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180404.514, now=131106-180605.372, channel=TCPChannel [type: server, contact: 127.0.0.1:41235]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 18:06:05,373-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180404.514, now=131106-180605.372, channel=TCPChannel [type: server, contact: 127.0.0.1:41235]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-06 18:06:05,373-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180404.514, now=131106-180605.372, channel=TCPChannel [type: server, contact: 127.0.0.1:41235]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 18:06:05,373-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180404.514, now=131106-180605.372, channel=TCPChannel [type: server, contact: 127.0.0.1:41235]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 18:06:05,373-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41235] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180404.514, now=131106-180605.372, channel=TCPChannel [type: server, contact: 127.0.0.1:41235]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 18:06:05,373-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41235] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180404.514, now=131106-180605.372, channel=TCPChannel [type: server, contact: 127.0.0.1:41235]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 18:06:05,373-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180404.715, now=131106-180605.573, channel=TCPChannel [type: server, contact: 127.0.0.1:41236]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 18:06:05,573-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180404.715, now=131106-180605.573, channel=TCPChannel [type: server, contact: 127.0.0.1:41236]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-06 18:06:05,573-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180404.715, now=131106-180605.573, channel=TCPChannel [type: server, contact: 127.0.0.1:41236]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 18:06:05,573-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180404.715, now=131106-180605.573, channel=TCPChannel [type: server, contact: 127.0.0.1:41236]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-06 18:06:05,573-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:41236] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180404.715, now=131106-180605.573, channel=TCPChannel [type: server, contact: 127.0.0.1:41236]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-06 18:06:05,574-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:41236] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131106-180404.715, now=131106-180605.573, channel=TCPChannel [type: server, contact: 127.0.0.1:41236]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-06 18:06:05,574-0600 INFO ChannelManager Channel already shut down
-No streams
-2013-11-06 18:06:14,758-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:06:24,768-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:06:34,777-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 18:06:36,958-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-06 18:06:40,432-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-No streams
-2013-11-06 18:06:44,786-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:06:54,795-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:07:04,803-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 18:07:13,432-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 18:07:14,812-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-06 18:07:17,750-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-No streams
-2013-11-06 18:07:24,822-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:07:34,831-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:07:44,841-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 18:07:49,906-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 18:07:54,850-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 18:07:55,068-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 18:08:04,859-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:08:14,868-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:08:24,876-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:08:26,379-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:08:32,385-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 18:08:34,884-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:08:44,892-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:08:54,900-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:09:02,854-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 18:09:04,908-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:09:09,703-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 18:09:14,917-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:09:24,927-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:09:34,929-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:09:39,329-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 18:09:44,933-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:09:47,021-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 18:09:54,937-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:10:04,954-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:10:14,963-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 18:10:15,803-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:10:24,339-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 18:10:24,972-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:10:34,981-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:10:44,991-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 18:10:52,277-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 18:10:55,000-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:11:01,657-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 18:11:05,009-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:11:15,019-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:11:25,027-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 18:11:28,751-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 18:11:35,036-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:11:38,975-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 18:11:45,046-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:11:55,056-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:12:05,065-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:12:05,224-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 18:12:15,073-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 18:12:16,294-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 18:12:25,082-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:12:35,092-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 18:12:41,699-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 18:12:45,101-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-06 18:12:53,612-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-No streams
-2013-11-06 18:12:55,109-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:13:05,121-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:13:15,132-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 18:13:18,173-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 18:13:25,141-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 18:13:30,930-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 18:13:35,150-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:13:45,159-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 18:13:54,647-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 18:13:55,167-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:14:05,178-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-06 18:14:08,248-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-No streams
-2013-11-06 18:14:15,188-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:14:25,198-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:14:31,120-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 18:14:35,207-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:14:45,216-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:14:45,566-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 18:14:55,224-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:15:05,233-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 18:15:07,595-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 18:15:15,241-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:15:22,884-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 18:15:25,250-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:15:35,258-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 18:15:44,069-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 18:15:45,266-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:15:55,274-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:16:00,202-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 18:16:05,286-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:16:15,295-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 18:16:20,543-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 18:16:25,303-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:16:35,311-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 18:16:37,521-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 18:16:45,324-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:16:55,335-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 18:16:57,017-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 18:17:05,345-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:17:14,838-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 18:17:15,354-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:17:25,363-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:17:33,490-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 18:17:35,371-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:17:45,380-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:17:52,156-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 18:17:55,389-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:18:05,398-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-2013-11-06 18:18:09,966-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-No streams
-2013-11-06 18:18:15,406-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:18:25,413-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:18:29,474-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 18:18:35,421-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:18:45,430-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 18:18:46,439-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 18:18:55,439-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:19:05,448-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 18:19:06,793-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 18:19:15,456-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:19:22,912-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 18:19:25,465-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:19:35,474-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:19:44,110-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 18:19:45,484-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:19:55,493-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:19:59,386-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 18:20:05,500-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:20:15,508-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:20:21,428-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 18:20:25,515-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:20:35,523-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:20:35,860-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 18:20:45,530-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:20:55,538-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:20:58,747-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 18:21:05,545-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 18:21:12,335-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 18:21:15,553-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:21:25,562-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:21:35,570-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:21:36,065-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 18:21:45,578-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 18:21:48,809-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 18:21:55,586-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:22:05,594-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 18:22:13,384-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 18:22:15,602-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:22:25,282-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 18:22:25,610-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:22:35,619-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:22:45,627-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 18:22:50,702-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 18:22:55,636-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 18:23:01,757-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 18:23:05,644-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:23:15,653-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:23:25,662-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 18:23:28,020-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 18:23:35,670-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:23:38,231-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 18:23:45,679-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:23:55,689-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:24:05,337-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 18:24:05,698-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 18:24:14,705-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 18:24:15,707-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:24:25,716-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:24:35,724-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:24:42,655-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 18:24:45,733-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-2013-11-06 18:24:51,180-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-No streams
-2013-11-06 18:24:55,742-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:25:05,750-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:25:15,759-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 18:25:19,974-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 18:25:25,767-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:25:27,652-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 18:25:35,776-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:25:45,786-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:25:55,794-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:25:57,291-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:26:04,126-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 18:26:05,802-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:26:15,810-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:26:25,818-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:26:34,609-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 18:26:35,828-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:26:40,600-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 18:26:45,836-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:26:55,845-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:27:05,856-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 18:27:11,928-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 18:27:15,867-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 18:27:17,075-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 18:27:25,876-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:27:35,886-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:27:45,895-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 18:27:49,246-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 18:27:53,549-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 18:27:55,905-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:28:05,914-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:28:15,924-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:28:25,933-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:28:26,564-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 18:28:30,023-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 18:28:35,943-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:28:45,953-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:28:55,962-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:29:03,882-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 18:29:05,972-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 18:29:06,497-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 18:29:15,982-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:29:25,992-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:29:36,001-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:29:41,200-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:29:42,970-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 18:29:46,011-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:29:56,020-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:30:06,028-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:30:16,036-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:30:18,518-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 18:30:19,445-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 18:30:26,045-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:30:36,054-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:30:46,061-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 18:30:55,837-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 18:30:55,919-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 18:30:56,070-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:31:06,080-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:31:16,088-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:31:26,097-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:31:32,392-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:31:33,154-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 18:31:36,106-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:31:46,114-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:31:56,123-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:32:06,131-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:32:08,866-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 18:32:10,473-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 18:32:16,140-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:32:26,148-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:32:36,156-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:32:45,340-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 18:32:46,165-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 18:32:47,791-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 18:32:56,173-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:33:06,181-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:33:16,189-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:33:21,814-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:33:25,108-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 18:33:26,197-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:33:36,205-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:33:46,212-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:33:56,220-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 18:33:58,289-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:34:02,426-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 18:34:06,228-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:34:16,236-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:34:26,244-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:34:34,762-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 18:34:36,252-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:34:39,744-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 18:34:46,260-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:34:56,268-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:35:06,276-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:35:11,236-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 18:35:16,284-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:35:17,062-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 18:35:26,292-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:35:36,299-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:35:46,307-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:35:47,710-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:35:54,380-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 18:35:56,315-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:36:06,323-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:36:16,331-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 18:36:24,185-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 18:36:26,339-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:36:31,698-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 18:36:36,347-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:36:46,355-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:36:56,363-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:37:00,658-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 18:37:06,371-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:37:09,016-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 18:37:16,379-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:37:26,387-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:37:36,395-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:37:37,132-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:37:46,334-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 18:37:46,403-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:37:56,411-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:38:06,419-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:38:13,606-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 18:38:16,427-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:38:23,652-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 18:38:26,435-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:38:36,443-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:38:46,451-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:38:50,080-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 18:38:56,459-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 18:39:00,969-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 18:39:06,467-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:39:16,475-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:39:26,483-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:39:26,554-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 18:39:36,491-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:39:38,288-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 18:39:46,499-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:39:56,507-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:40:03,028-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 18:40:06,515-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:40:15,606-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 18:40:16,523-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:40:26,531-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:40:36,539-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 18:40:39,503-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 18:40:46,547-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:40:52,924-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 18:40:56,555-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:41:06,563-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:41:15,976-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 18:41:16,571-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:41:26,579-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:41:30,242-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 18:41:36,587-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:41:46,595-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:41:52,450-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 18:41:56,602-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:42:06,610-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:42:07,560-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 18:42:16,618-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:42:26,626-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:42:28,924-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 18:42:36,634-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:42:44,878-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 18:42:46,642-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:42:56,650-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:43:05,399-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 18:43:06,657-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:43:16,665-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 18:43:22,195-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 18:43:26,673-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:43:36,681-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:43:41,873-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 18:43:46,688-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:43:56,696-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:43:59,514-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 18:44:06,704-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:44:16,712-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:44:18,347-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 18:44:26,720-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:44:36,728-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:44:36,832-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 18:44:46,735-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:44:54,821-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 18:44:56,743-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:45:06,751-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:45:14,150-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 18:45:16,759-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:45:26,767-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 18:45:31,296-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 18:45:36,775-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:45:46,783-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:45:51,468-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 18:45:56,791-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:46:06,798-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:46:07,769-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 18:46:16,806-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:46:26,814-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:46:28,786-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 18:46:36,822-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:46:44,243-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 18:46:46,830-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:46:56,838-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:47:06,104-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 18:47:06,846-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:47:16,853-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 18:47:20,718-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 18:47:26,861-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:47:36,869-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:47:43,423-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 18:47:46,877-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:47:56,885-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:47:57,191-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 18:48:06,893-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:48:16,901-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:48:20,741-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 18:48:26,908-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:48:33,665-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 18:48:36,916-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:48:46,924-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:48:56,932-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:48:58,059-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 18:49:06,940-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:49:10,139-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 18:49:16,948-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:49:26,955-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:49:35,377-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 18:49:36,963-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:49:46,614-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 18:49:46,971-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:49:56,979-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:50:06,987-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:50:12,695-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 18:50:16,994-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:50:23,088-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 18:50:27,002-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:50:37,010-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:50:47,018-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 18:50:50,014-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 18:50:57,026-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:50:59,562-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 18:51:07,034-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:51:17,041-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:51:27,049-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:51:27,331-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:51:36,036-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 18:51:37,057-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:51:47,065-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:51:57,073-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:52:04,649-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 18:52:07,081-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:52:12,510-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 18:52:17,088-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:52:27,096-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:52:37,104-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:52:41,967-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 18:52:47,112-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:52:48,984-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 18:52:57,120-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:53:07,128-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:53:17,136-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:53:19,286-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 18:53:25,459-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 18:53:27,143-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:53:37,151-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:53:47,159-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:53:56,604-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 18:53:57,167-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:54:01,932-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 18:54:07,175-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:54:17,183-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:54:27,191-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:54:33,922-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 18:54:37,198-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:54:38,406-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 18:54:47,206-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:54:57,214-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:55:07,222-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:55:11,240-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:55:14,880-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 18:55:17,229-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:55:27,237-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:55:37,245-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:55:47,253-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:55:48,558-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 18:55:51,355-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 18:55:57,261-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:56:07,269-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:56:17,277-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 18:56:25,876-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 18:56:27,284-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 18:56:27,829-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 18:56:37,292-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:56:47,300-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:56:57,308-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:57:03,195-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:57:04,302-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 18:57:07,316-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:57:17,324-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:57:27,331-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:57:37,339-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:57:40,513-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:57:40,776-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 18:57:47,347-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:57:57,355-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:58:07,363-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 18:58:17,251-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 18:58:17,371-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 18:58:17,831-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 18:58:27,378-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:58:37,386-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:58:47,394-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:58:53,724-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:58:55,150-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 18:58:57,402-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:59:07,410-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:59:17,418-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:59:27,426-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 18:59:30,198-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 18:59:32,468-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 18:59:37,433-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:59:47,441-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 18:59:57,449-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:00:06,672-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:00:07,457-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:00:09,786-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:00:17,465-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:00:27,473-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:00:37,481-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:00:43,147-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:00:47,104-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:00:47,489-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:00:57,497-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:01:07,505-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:01:17,513-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:01:19,621-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:01:24,422-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:01:27,520-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:01:37,528-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:01:47,536-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:01:56,095-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:01:57,544-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:02:01,740-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:02:07,552-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:02:17,560-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:02:27,568-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:02:32,569-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:02:37,576-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:02:39,058-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:02:47,583-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:02:57,591-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:03:07,599-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 19:03:09,044-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:03:16,376-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:03:17,607-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:03:27,615-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:03:37,623-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:03:45,517-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:03:47,631-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:03:53,694-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:03:57,639-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:04:07,647-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:04:17,655-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:04:21,991-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:04:27,663-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:04:31,012-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:04:37,672-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:04:47,680-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:04:57,688-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 19:04:58,465-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 19:05:07,696-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:05:08,330-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:05:17,704-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:05:27,712-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 19:05:34,941-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 19:05:37,720-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:05:45,648-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:05:47,729-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:05:57,737-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:06:07,745-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:06:11,414-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:06:17,753-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 19:06:22,966-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 19:06:27,761-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:06:37,769-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:06:47,777-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:06:47,888-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:06:57,785-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:07:00,285-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:07:07,793-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:07:17,801-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:07:24,362-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:07:27,809-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:07:37,603-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:07:37,817-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:07:47,825-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:07:57,833-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:08:00,836-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:08:07,841-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 19:08:14,920-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 19:08:17,849-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:08:27,857-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:08:37,310-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:08:37,865-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:08:47,873-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:08:52,239-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:08:57,881-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:09:07,889-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:09:13,784-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:09:17,897-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:09:27,905-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:09:29,557-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:09:37,913-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:09:47,920-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 19:09:50,259-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 19:09:57,928-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 19:10:06,875-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 19:10:07,936-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:10:17,944-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:10:26,732-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:10:27,952-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:10:37,960-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:10:44,194-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:10:47,968-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:10:57,976-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:11:03,206-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:11:07,984-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:11:17,992-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:11:21,512-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:11:28,000-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:11:38,009-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:11:39,680-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:11:48,017-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:11:58,025-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:11:58,830-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:12:08,033-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:12:16,155-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:12:18,041-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:12:28,050-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:12:36,148-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:12:38,058-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:12:48,065-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:12:52,629-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:12:58,073-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:13:08,081-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:13:13,466-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:13:18,089-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:13:28,098-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:13:29,103-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:13:38,106-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:13:48,114-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:13:50,784-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:13:58,122-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:14:05,577-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:14:08,130-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:14:18,139-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 19:14:28,101-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 19:14:28,147-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:14:38,155-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:14:42,051-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:14:48,163-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:14:58,172-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:15:05,420-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:15:08,180-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:15:18,188-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:15:18,525-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:15:28,196-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:15:38,205-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:15:42,738-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:15:48,213-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:15:54,999-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:15:58,221-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:16:08,229-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:16:18,237-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:16:20,056-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:16:28,246-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 19:16:31,474-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 19:16:38,254-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:16:48,262-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:16:57,374-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:16:58,270-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 19:17:07,948-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 19:17:08,278-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:17:18,287-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:17:28,295-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 19:17:34,691-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 19:17:38,303-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:17:44,421-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:17:48,311-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:17:58,319-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:18:08,328-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:18:12,010-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:18:18,336-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:18:20,895-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:18:28,344-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:18:38,352-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:18:48,361-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:18:49,328-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 19:18:57,370-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 19:18:58,369-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:19:08,377-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:19:18,386-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:19:26,646-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:19:28,394-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:19:33,843-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:19:38,402-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:19:48,410-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:19:58,419-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:20:03,964-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:20:08,427-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:20:10,317-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:20:18,435-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:20:28,443-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:20:38,451-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:20:41,282-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 19:20:46,792-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 19:20:48,459-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:20:58,467-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:21:08,475-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:21:18,483-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:21:18,600-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:21:23,265-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:21:28,490-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:21:38,498-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:21:48,506-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:21:55,918-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:21:58,514-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:21:59,739-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:22:08,522-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:22:18,530-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:22:28,538-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:22:33,236-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:22:36,213-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:22:38,546-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:22:48,555-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:22:58,563-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:23:08,571-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:23:10,554-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:23:12,687-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:23:18,579-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:23:28,587-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:23:38,595-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:23:47,872-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:23:48,604-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:23:49,161-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:23:58,612-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:24:08,620-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:24:18,628-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:24:25,190-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:24:25,635-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:24:28,636-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:24:38,644-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:24:48,652-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:24:58,660-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:25:02,110-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:25:02,508-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:25:08,669-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:25:18,677-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:25:28,685-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 19:25:38,584-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 19:25:38,693-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:25:39,826-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:25:48,701-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:25:58,709-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:26:08,717-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:26:15,057-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:26:17,144-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:26:18,726-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:26:28,734-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:26:38,742-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:26:48,750-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:26:51,531-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:26:54,462-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:26:58,758-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:27:08,766-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:27:18,774-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:27:28,005-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:27:28,782-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:27:31,780-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:27:38,789-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:27:48,797-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:27:58,805-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:28:04,479-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:28:08,813-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:28:09,098-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:28:18,821-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:28:28,829-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:28:38,837-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:28:40,953-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:28:46,416-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:28:48,844-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:28:58,852-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:29:08,860-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:29:17,427-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:29:18,868-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:29:23,734-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:29:28,876-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:29:38,884-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:29:48,892-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:29:53,901-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:29:58,900-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 19:30:01,051-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 19:30:08,907-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:30:18,915-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:30:28,923-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 19:30:30,376-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:30:38,370-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:30:38,931-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:30:48,939-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:30:58,947-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:31:06,849-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:31:08,955-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:31:15,688-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:31:18,962-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:31:28,970-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:31:38,978-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:31:43,323-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:31:48,986-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 19:31:53,006-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 19:31:58,994-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:32:09,002-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:32:19,009-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 19:32:19,798-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 19:32:29,017-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:32:30,325-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:32:39,025-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:32:49,033-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:32:56,271-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:32:59,041-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:33:07,643-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:33:09,049-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:33:19,057-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:33:29,064-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:33:32,745-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:33:39,072-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:33:44,961-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:33:49,080-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:33:59,088-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:34:09,096-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:34:09,219-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:34:19,104-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:34:22,279-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:34:29,111-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:34:39,119-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 19:34:45,694-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 19:34:49,127-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:34:59,135-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:34:59,597-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:35:09,144-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:35:19,152-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 19:35:22,168-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 19:35:29,160-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 19:35:36,916-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 19:35:39,168-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:35:49,177-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:35:58,641-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:35:59,185-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:36:09,193-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:36:14,233-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:36:19,201-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:36:29,209-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:36:35,115-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:36:39,217-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:36:49,225-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:36:51,551-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:36:59,233-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:37:09,241-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:37:11,589-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:37:19,249-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:37:28,869-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:37:29,257-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:37:39,265-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:37:48,063-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:37:49,273-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:37:59,281-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:38:06,187-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:38:09,289-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:38:19,298-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:38:24,537-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:38:29,306-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:38:39,314-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 19:38:43,505-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 19:38:49,322-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:38:59,330-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:39:01,011-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:39:09,338-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:39:19,346-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:39:20,824-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:39:29,354-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:39:37,485-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:39:39,362-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:39:49,370-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:39:58,142-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:39:59,377-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:40:09,385-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:40:13,959-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:40:19,393-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:40:29,401-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 19:40:35,461-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 19:40:39,409-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:40:49,417-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:40:50,433-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:40:59,425-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:41:09,432-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:41:12,778-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:41:19,440-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 19:41:26,908-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 19:41:29,448-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:41:39,456-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:41:49,464-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 19:41:50,095-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 19:41:59,472-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:42:03,381-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:42:09,480-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:42:19,488-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:42:27,414-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:42:29,496-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:42:39,504-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:42:39,855-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:42:49,511-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:42:59,519-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:43:04,732-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:43:09,527-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:43:16,329-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:43:19,535-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:43:29,543-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:43:39,551-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:43:42,050-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:43:49,558-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:43:52,803-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:43:59,566-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:44:09,574-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:44:19,368-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:44:19,582-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:44:29,277-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:44:29,589-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:44:39,597-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:44:49,605-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:44:56,686-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:44:59,613-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 19:45:05,752-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 19:45:09,620-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:45:19,628-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:45:29,636-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:45:34,004-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:45:39,644-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:45:42,225-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:45:49,651-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:45:59,659-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:46:09,667-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:46:11,322-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:46:18,699-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:46:19,674-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:46:29,682-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:46:39,690-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:46:48,640-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:46:49,698-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:46:55,173-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:46:59,705-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:47:09,713-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:47:19,721-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:47:25,958-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:47:29,728-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 19:47:31,647-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 19:47:39,736-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:47:49,744-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:47:59,752-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:48:03,276-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 19:48:08,123-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 19:48:09,759-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:48:19,767-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:48:29,775-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:48:39,783-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:48:40,594-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:48:44,596-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:48:49,790-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:48:59,798-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:49:09,806-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:49:17,912-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:49:19,814-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:49:21,070-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:49:29,821-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:49:39,829-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:49:49,837-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:49:55,230-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:49:57,544-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:49:59,844-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:50:09,852-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:50:19,860-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:50:29,868-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:50:32,549-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:50:34,018-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:50:39,875-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:50:49,883-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:50:59,891-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:51:09,867-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:51:09,898-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:51:10,492-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:51:19,906-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:51:29,914-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:51:39,922-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:51:46,966-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:51:47,185-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:51:49,929-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:51:59,937-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:52:09,945-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:52:19,953-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:52:23,440-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:52:24,503-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:52:29,960-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:52:39,968-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:52:49,976-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:52:59,914-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:52:59,984-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:53:01,821-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:53:09,991-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:53:19,999-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:53:30,007-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:53:36,388-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:53:39,139-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:53:40,015-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:53:50,022-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:54:00,030-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:54:10,038-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:54:12,862-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:54:16,457-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:54:20,046-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:54:30,053-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:54:40,061-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 19:54:49,337-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 19:54:50,069-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 19:54:53,775-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 19:55:00,076-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:55:10,084-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:55:20,092-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 19:55:25,811-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 19:55:30,100-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:55:31,094-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:55:40,108-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:55:50,115-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:56:00,123-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:56:02,284-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:56:08,412-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:56:10,131-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:56:20,138-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:56:30,146-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:56:38,758-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:56:40,154-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:56:45,730-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:56:50,162-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:57:00,169-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:57:10,177-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:57:15,232-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:57:20,185-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:57:23,048-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:57:30,193-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:57:40,200-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:57:50,208-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:57:51,706-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:58:00,216-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 19:58:00,366-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 19:58:10,224-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:58:20,232-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:58:28,180-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:58:30,239-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:58:37,684-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:58:40,247-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:58:50,255-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:59:00,263-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:59:04,654-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 19:59:10,270-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 19:59:15,001-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 19:59:20,278-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:59:30,286-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 19:59:40,294-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 19:59:41,128-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 19:59:50,301-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 19:59:52,320-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:00:00,309-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:00:10,317-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:00:17,602-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 20:00:20,325-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:00:29,638-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:00:30,332-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:00:40,340-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:00:50,348-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:00:54,076-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:01:00,356-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:01:06,956-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 20:01:10,364-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:01:20,371-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:01:30,379-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:01:30,550-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:01:40,387-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:01:44,274-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:01:50,395-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:02:00,403-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:02:07,024-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:02:10,410-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:02:20,418-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:02:21,592-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 20:02:30,426-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:02:40,434-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:02:43,498-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 20:02:50,442-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:02:58,910-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:03:00,449-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:03:10,457-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 20:03:19,971-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 20:03:20,465-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:03:30,473-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 20:03:36,228-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 20:03:40,480-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:03:50,488-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 20:03:56,447-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 20:04:00,496-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:04:10,504-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:04:13,547-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:04:20,512-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:04:30,520-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:04:32,920-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 20:04:40,527-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:04:50,535-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:04:50,865-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:05:00,543-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:05:09,394-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:05:10,551-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:05:20,559-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:05:28,183-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 20:05:30,566-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:05:40,574-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 20:05:45,869-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 20:05:50,582-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:06:00,590-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:06:05,501-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:06:10,597-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:06:20,605-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:06:22,342-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:06:30,613-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:06:40,621-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:06:42,819-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 20:06:50,629-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:06:58,816-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 20:07:00,636-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:07:10,644-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:07:20,137-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:07:20,652-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:07:30,660-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:07:35,290-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:07:40,668-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:07:50,676-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 20:07:57,454-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 20:08:00,683-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:08:10,691-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:08:11,764-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:08:20,699-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:08:30,707-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:08:34,773-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:08:40,715-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:08:48,238-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:08:50,722-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:09:00,730-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:09:10,738-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:09:12,091-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:09:20,746-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 20:09:24,713-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 20:09:30,754-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:09:40,762-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:09:49,409-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 20:09:50,769-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:10:00,777-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:10:01,186-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:10:10,785-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:10:20,793-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:10:26,727-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:10:30,801-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 10ms
-2013-11-06 20:10:37,669-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 10ms
-No streams
-2013-11-06 20:10:40,809-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:10:50,816-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:11:00,824-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 20:11:04,044-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 20:11:10,832-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:11:14,134-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:11:20,840-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:11:30,848-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:11:40,856-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:11:41,363-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:11:50,608-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 20:11:50,863-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:12:00,871-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:12:10,879-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:12:18,681-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:12:20,887-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:12:27,082-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:12:30,895-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:12:40,903-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:12:50,910-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:12:55,999-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:13:00,918-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:13:03,556-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:13:10,926-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:13:20,934-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:13:30,942-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:13:33,317-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:13:40,030-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:13:40,950-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:13:50,957-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:14:00,965-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:14:10,635-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 20:14:10,973-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:14:16,504-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:14:20,981-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:14:30,989-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:14:40,996-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:14:47,953-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:14:51,004-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 20:14:52,979-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 20:15:01,012-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:15:11,020-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:15:21,028-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:15:25,271-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 20:15:29,453-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 20:15:31,036-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:15:41,043-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:15:51,051-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:16:01,059-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:16:02,589-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:16:05,926-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:16:11,067-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:16:21,075-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:16:31,083-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:16:39,907-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:16:41,090-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:16:42,400-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:16:51,098-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:17:01,106-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:17:11,114-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:17:17,225-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:17:18,874-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 20:17:21,122-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:17:31,129-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:17:41,137-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:17:51,145-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:17:54,543-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:17:55,348-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 20:18:01,153-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:18:11,161-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:18:21,169-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:18:31,176-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:18:31,822-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 20:18:31,860-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 20:18:41,184-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:18:51,192-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:19:01,200-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:19:08,296-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:19:09,179-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:19:11,208-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:19:21,215-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:19:31,223-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:19:41,231-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:19:44,770-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:19:46,497-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:19:51,239-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:20:01,247-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:20:11,255-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:20:21,244-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:20:21,262-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 20:20:23,816-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 20:20:31,270-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:20:41,278-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:20:51,286-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 20:20:57,719-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:21:01,133-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:21:01,294-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:21:11,302-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:21:21,310-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:21:31,318-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 20:21:34,193-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:21:38,451-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:21:41,325-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:21:51,333-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:22:01,341-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 20:22:10,667-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 20:22:11,349-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:22:15,769-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:22:21,357-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:22:31,364-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:22:41,372-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 20:22:47,141-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 20:22:51,380-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:22:53,087-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:23:01,388-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:23:11,395-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:23:21,403-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 20:23:23,615-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:23:30,405-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 20:23:31,410-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:23:41,419-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:23:51,427-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 20:24:00,089-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 20:24:01,435-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:24:07,723-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:24:11,443-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:24:21,450-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:24:31,458-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 20:24:36,563-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 20:24:41,466-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:24:45,041-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:24:51,474-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:25:01,481-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:25:11,489-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:25:13,036-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:25:21,497-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:25:22,359-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:25:31,505-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:25:41,513-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 20:25:49,511-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 20:25:51,520-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:25:59,677-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:26:01,529-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:26:11,537-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:26:21,545-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-2013-11-06 20:26:25,986-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-No streams
-2013-11-06 20:26:31,553-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:26:36,995-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:26:41,561-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:26:51,569-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:27:01,577-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:27:02,458-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:27:11,585-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:27:14,313-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 20:27:21,593-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:27:31,601-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 20:27:38,933-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 20:27:41,609-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:27:51,617-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:27:51,631-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:28:01,625-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:28:11,633-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:28:15,406-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:28:21,641-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:28:28,950-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:28:31,648-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:28:41,656-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:28:51,664-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:28:51,880-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:29:01,672-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:29:06,268-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:29:11,680-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:29:21,688-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:29:28,354-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:29:31,696-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:29:41,704-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:29:43,586-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:29:51,711-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:30:01,719-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:30:04,828-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 20:30:11,727-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 20:30:20,905-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 20:30:21,735-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:30:31,743-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:30:41,302-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:30:41,750-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:30:51,758-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:30:58,222-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:31:01,766-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:31:11,773-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:31:17,777-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 20:31:21,781-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:31:31,789-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:31:35,540-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:31:41,797-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:31:51,805-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:31:54,250-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:32:01,813-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:32:11,821-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:32:12,858-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:32:21,829-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:32:30,724-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:32:31,837-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:32:41,844-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:32:50,176-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:32:51,852-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:33:01,860-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:33:07,198-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:33:11,868-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:33:21,875-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:33:27,494-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 20:33:31,883-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:33:41,891-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:33:43,672-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 20:33:51,898-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:34:01,906-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:34:04,812-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:34:11,914-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:34:20,146-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:34:21,922-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:34:31,930-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:34:41,938-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:34:42,130-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 20:34:51,946-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:34:56,621-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 20:35:01,954-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:35:11,962-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:35:19,448-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 20:35:21,970-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:35:31,978-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:35:33,094-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 20:35:41,986-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:35:51,994-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:35:56,766-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:36:02,003-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:36:09,568-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 20:36:12,011-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:36:22,019-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:36:32,027-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:36:34,084-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 20:36:42,036-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:36:46,042-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:36:52,044-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:37:02,052-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:37:11,402-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:37:12,060-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:37:22,069-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:37:22,516-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 20:37:32,077-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:37:42,085-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:37:48,720-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:37:52,094-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:37:58,990-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:38:02,102-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:38:12,110-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:38:22,118-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:38:26,038-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 20:38:32,126-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:38:35,464-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 20:38:42,135-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:38:52,143-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:39:02,151-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:39:03,356-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:39:11,938-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:39:12,159-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:39:22,168-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:39:32,176-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:39:40,674-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 20:39:42,184-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:39:48,412-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 20:39:52,192-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:40:02,200-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:40:12,208-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:40:17,992-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 20:40:22,216-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 20:40:24,887-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 20:40:32,224-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:40:42,232-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:40:52,241-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:40:55,310-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:41:01,360-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:41:02,249-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:41:12,257-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:41:22,264-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:41:32,272-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:41:32,628-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:41:37,834-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:41:42,280-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:41:52,288-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:42:02,296-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:42:09,946-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 20:42:12,304-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:42:14,308-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:42:22,312-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:42:32,320-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:42:42,328-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:42:47,264-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:42:50,782-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 20:42:52,336-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:43:02,344-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:43:12,352-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:43:22,360-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:43:24,582-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:43:27,256-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:43:32,367-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:43:42,375-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:43:52,383-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:44:01,900-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:44:02,391-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:44:03,730-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:44:12,399-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:44:22,407-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:44:32,415-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 20:44:39,218-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:44:40,204-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:44:42,423-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:44:52,431-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:45:02,439-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:45:12,447-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:45:16,537-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:45:16,678-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 20:45:22,455-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:45:32,463-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:45:42,471-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:45:52,479-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:45:53,152-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:45:53,855-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 20:46:02,486-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:46:12,494-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:46:22,502-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:46:29,626-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 20:46:31,172-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 20:46:32,509-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:46:42,517-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:46:52,525-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:47:02,532-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:47:06,100-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:47:08,491-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 20:47:12,540-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:47:22,548-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:47:32,556-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:47:42,565-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:47:42,574-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:47:45,809-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:47:52,573-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:48:02,581-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:48:12,589-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 20:48:19,047-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 20:48:22,597-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:48:23,127-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 20:48:32,604-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:48:42,612-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:48:52,619-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:48:55,522-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:49:00,445-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:49:02,627-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:49:12,635-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:49:22,644-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:49:31,996-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:49:32,652-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:49:37,763-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 20:49:42,660-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:49:52,668-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:50:02,677-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:50:08,470-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:50:12,685-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:50:15,081-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:50:22,693-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:50:32,701-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:50:42,709-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:50:44,945-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 20:50:52,399-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 20:50:52,718-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:51:02,726-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:51:12,734-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:51:21,418-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 20:51:22,742-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:51:29,718-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 20:51:32,751-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:51:42,759-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:51:52,767-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:51:57,892-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 20:52:02,775-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:52:07,036-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:52:12,783-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:52:22,791-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:52:32,800-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:52:34,366-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:52:42,808-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:52:44,354-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 20:52:52,816-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:53:02,825-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 20:53:10,841-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 20:53:12,833-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:53:21,672-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 20:53:22,841-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:53:32,849-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:53:42,857-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:53:47,314-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 20:53:52,864-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 20:53:58,990-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 20:54:02,872-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:54:12,880-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:54:22,888-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:54:23,788-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 20:54:32,895-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:54:36,309-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 20:54:42,903-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:54:52,911-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:55:00,262-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 20:55:02,918-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:55:12,926-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:55:13,627-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:55:22,934-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:55:32,942-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 20:55:36,738-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 20:55:42,950-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:55:50,945-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 20:55:52,958-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:56:02,965-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:56:12,973-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:56:13,211-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 20:56:22,981-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:56:28,263-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 20:56:32,989-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:56:42,997-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 20:56:49,685-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 20:56:53,005-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:57:03,012-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 20:57:05,581-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 20:57:13,020-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:57:23,028-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:57:26,160-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 20:57:33,036-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:57:42,900-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 20:57:43,044-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:57:53,052-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:58:02,634-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 20:58:03,059-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:58:13,067-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 20:58:20,217-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 20:58:23,075-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:58:33,083-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:58:39,108-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:58:43,091-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:58:53,099-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:58:57,536-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 20:59:03,107-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:59:13,115-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:59:15,582-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 20:59:23,123-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 20:59:33,130-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 20:59:34,854-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 20:59:43,138-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 20:59:52,056-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 20:59:53,146-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:00:03,154-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:00:12,172-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:00:13,162-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:00:23,170-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 21:00:28,531-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 21:00:33,178-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:00:43,185-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:00:49,490-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:00:53,193-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:01:03,201-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 21:01:05,004-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 21:01:13,208-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:01:23,216-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 21:01:26,808-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 21:01:33,224-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:01:41,478-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:01:43,232-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:01:53,239-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:02:03,247-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:02:04,126-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:02:13,255-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 21:02:17,953-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 21:02:23,263-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:02:33,271-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 21:02:41,444-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 21:02:43,279-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:02:53,286-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:02:54,426-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:03:03,294-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:03:13,302-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:03:18,762-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:03:23,310-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 21:03:30,900-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 21:03:33,318-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:03:43,326-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:03:53,334-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 21:03:56,080-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 21:04:03,342-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:04:07,374-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:04:13,351-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:04:23,359-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:04:33,367-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:04:33,398-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:04:43,375-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 21:04:43,847-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 21:04:53,383-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:05:03,391-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:05:10,716-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:05:13,399-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:05:20,322-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:05:23,407-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:05:33,416-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:05:43,424-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 21:05:48,035-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 21:05:53,432-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 21:05:56,797-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 21:06:03,440-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:06:13,448-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:06:23,456-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:06:25,352-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:06:33,271-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:06:33,464-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:06:43,471-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:06:53,479-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:07:02,670-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:07:03,487-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:07:09,745-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:07:13,495-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:07:23,502-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:07:33,510-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:07:39,988-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:07:43,518-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:07:46,219-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:07:53,525-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:08:03,533-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:08:13,541-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 21:08:17,306-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:08:22,693-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:08:23,549-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:08:33,557-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:08:43,564-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:08:53,572-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:08:54,624-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 21:08:59,167-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 21:09:03,580-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:09:13,587-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:09:23,595-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:09:31,942-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:09:33,603-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:09:35,641-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:09:43,611-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:09:53,619-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:10:03,627-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:10:09,260-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:10:12,115-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:10:13,634-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:10:23,642-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:10:33,650-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:10:43,658-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:10:46,578-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:10:48,589-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:10:53,665-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:11:03,673-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:11:13,681-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:11:23,689-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 21:11:23,895-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 21:11:25,063-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 21:11:33,697-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:11:43,704-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:11:53,712-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:12:01,214-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:12:01,537-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:12:03,720-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:12:13,728-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:12:23,736-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:12:33,743-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 21:12:38,011-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:12:38,532-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:12:43,751-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:12:53,759-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:13:03,767-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:13:13,775-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:13:14,485-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:13:15,850-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:13:23,783-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:13:33,790-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:13:43,798-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 21:13:50,959-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:13:53,168-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:13:53,806-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:14:03,814-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:14:13,822-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:14:23,829-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:14:27,434-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:14:30,486-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:14:33,837-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:14:43,845-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:14:53,853-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:15:03,861-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 21:15:03,909-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:15:07,804-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:15:13,869-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:15:23,876-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:15:33,884-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 21:15:40,383-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 21:15:43,892-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:15:45,122-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:15:53,900-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:16:03,908-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:16:13,916-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 21:16:16,856-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:16:22,440-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:16:23,923-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:16:33,931-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:16:43,939-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:16:53,330-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:16:53,947-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:16:59,758-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:17:03,954-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:17:13,962-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:17:23,970-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 21:17:29,805-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 21:17:33,978-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:17:37,076-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:17:43,986-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:17:53,994-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:18:04,001-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:18:06,278-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:18:14,009-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:18:14,394-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:18:24,017-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:18:34,025-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:18:42,752-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:18:44,033-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:18:51,712-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:18:54,040-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:19:04,048-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:19:14,056-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:19:19,226-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:19:24,063-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:19:29,030-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:19:34,071-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:19:44,079-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:19:54,087-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:19:55,700-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:20:04,095-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 21:20:06,347-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 21:20:14,103-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:20:24,111-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:20:32,174-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:20:34,119-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:20:43,666-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:20:44,127-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:20:54,135-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:21:04,142-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:21:08,648-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:21:14,150-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:21:20,984-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:21:24,158-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:21:34,166-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:21:44,174-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 21:21:45,121-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 21:21:54,182-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:21:58,302-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:22:04,190-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:22:14,198-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:22:21,596-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:22:24,206-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:22:34,214-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:22:35,620-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:22:44,222-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:22:54,230-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 21:22:58,070-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 21:23:04,238-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 21:23:12,938-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 21:23:14,246-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:23:24,254-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:23:34,262-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:23:34,544-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:23:44,269-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:23:50,257-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:23:54,277-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:24:04,285-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 21:24:11,018-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 21:24:14,293-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:24:24,300-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 21:24:27,574-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 21:24:34,308-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:24:44,316-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 21:24:47,493-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 21:24:54,323-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:25:04,331-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:25:04,893-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:25:14,339-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:25:23,967-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:25:24,347-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:25:34,354-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:25:42,211-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:25:44,362-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:25:54,370-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:26:00,441-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:26:04,378-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:26:14,386-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:26:19,529-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:26:24,394-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:26:34,402-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:26:36,915-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:26:44,410-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:26:54,418-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:26:56,847-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:27:04,426-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:27:13,389-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:27:14,434-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:27:24,442-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:27:34,165-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:27:34,449-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:27:44,457-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:27:49,863-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:27:54,465-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:28:04,473-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:28:11,483-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:28:14,481-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:28:24,488-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:28:26,337-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:28:34,496-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:28:44,504-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:28:48,801-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:28:54,512-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:29:02,811-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:29:04,520-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:29:14,528-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:29:24,536-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:29:26,119-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:29:34,543-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 21:29:39,284-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 21:29:44,551-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:29:54,559-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 21:30:03,436-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 21:30:04,567-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:30:14,574-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 21:30:15,760-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 21:30:24,582-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:30:34,590-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:30:40,755-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:30:44,598-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:30:52,233-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:30:54,605-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:31:04,613-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:31:14,621-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:31:18,073-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:31:24,629-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 21:31:28,708-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 21:31:34,637-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:31:44,644-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:31:54,652-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:31:55,391-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:32:04,660-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 21:32:05,181-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 21:32:14,668-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:32:24,676-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:32:32,709-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:32:34,683-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:32:41,655-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:32:44,691-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:32:54,699-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:33:04,707-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:33:10,027-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:33:14,715-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:33:18,130-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:33:24,723-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:33:34,731-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:33:44,738-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:33:47,345-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:33:54,604-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:33:54,746-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:34:04,754-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:34:14,762-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-06 21:34:24,662-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-06 21:34:24,770-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 21:34:31,079-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 21:34:34,778-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:34:44,786-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:34:54,794-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:35:01,981-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:35:04,802-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:35:07,553-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:35:14,810-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:35:24,817-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:35:34,825-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:35:39,299-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:35:44,027-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:35:44,833-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:35:54,841-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:36:04,849-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:36:14,857-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:36:16,617-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:36:20,501-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:36:24,865-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:36:34,873-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:36:44,881-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:36:53,935-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:36:54,889-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 21:36:56,974-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 21:37:04,897-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:37:14,905-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:37:24,913-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:37:31,253-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:37:33,449-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:37:34,921-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:37:44,929-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:37:54,936-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:38:04,944-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:38:08,571-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:38:09,924-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:38:14,951-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:38:24,959-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:38:34,966-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:38:44,974-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:38:45,889-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:38:46,398-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:38:54,982-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:39:04,990-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:39:14,998-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 21:39:22,873-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-06 21:39:23,207-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-06 21:39:25,006-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:39:35,014-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:39:45,022-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:39:55,030-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:39:59,346-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:40:00,526-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:40:05,038-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:40:15,046-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:40:25,053-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:40:35,061-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 21:40:35,821-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:40:37,844-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:40:45,070-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:40:55,078-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:41:05,086-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:41:12,294-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:41:15,093-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:41:15,162-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:41:25,101-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:41:35,109-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:41:45,116-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 21:41:48,769-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:41:52,480-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:41:55,124-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:42:05,132-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:42:15,139-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:42:25,147-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 3ms
-2013-11-06 21:42:25,244-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 3ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:42:29,798-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:42:35,154-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:42:45,162-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:42:55,169-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:43:01,716-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:43:05,177-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:43:07,116-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:43:15,184-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:43:25,192-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:43:35,199-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:43:38,190-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:43:44,434-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:43:45,207-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:43:55,215-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:44:05,223-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:44:14,664-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:44:15,231-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:44:21,752-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:44:25,239-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:44:35,246-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:44:45,254-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:44:51,138-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:44:55,262-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 21:44:59,070-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 21:45:05,270-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:45:15,278-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:45:25,285-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 21:45:27,613-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 21:45:35,293-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:45:36,388-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:45:45,301-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:45:55,309-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 21:46:04,087-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 21:46:05,316-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:46:13,706-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:46:15,324-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:46:25,331-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:46:35,339-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:46:40,560-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:46:45,347-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:46:51,024-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:46:55,354-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:47:05,362-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:47:15,369-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:47:17,034-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:47:25,377-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:47:28,342-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:47:35,385-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:47:45,393-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:47:53,508-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:47:55,401-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:48:05,409-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:48:05,660-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:48:15,417-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:48:25,424-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 21:48:29,983-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 21:48:35,432-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:48:42,978-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:48:45,440-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:48:55,448-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:49:05,456-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:49:06,456-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:49:15,464-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:49:20,297-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:49:25,472-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:49:35,480-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:49:42,930-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:49:45,488-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:49:55,495-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:49:57,615-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:50:05,503-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:50:15,511-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:50:19,404-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:50:25,519-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:50:34,933-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:50:35,527-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:50:45,535-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:50:55,543-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 21:50:55,879-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 21:51:05,551-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 21:51:12,251-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 21:51:15,559-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:51:25,567-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:51:32,352-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:51:35,575-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:51:45,583-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:51:49,569-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:51:55,591-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:52:05,599-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 21:52:08,826-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 21:52:15,607-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:52:25,615-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 21:52:26,887-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 21:52:35,623-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:52:45,300-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:52:45,631-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:52:55,640-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:53:04,205-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:53:05,648-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:53:15,656-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:53:21,774-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:53:25,664-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:53:35,672-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-06 21:53:41,522-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-06 21:53:45,681-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:53:55,689-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:53:58,248-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:54:05,697-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:54:15,705-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:54:18,841-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:54:25,713-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:54:34,722-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:54:35,721-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:54:45,729-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:54:55,737-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:54:56,159-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:55:05,746-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:55:11,196-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:55:15,754-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:55:25,762-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 21:55:33,478-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 21:55:35,770-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:55:45,778-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 21:55:47,670-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 21:55:55,786-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:56:05,794-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:56:10,795-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:56:15,803-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:56:24,144-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:56:25,811-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:56:35,819-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:56:45,827-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 21:56:48,113-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 21:56:55,835-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:57:00,618-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:57:05,843-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:57:15,851-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:57:25,431-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:57:25,859-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:57:35,867-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 21:57:37,093-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 21:57:45,876-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:57:55,884-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 21:58:02,749-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 21:58:05,892-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:58:13,566-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:58:15,900-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:58:25,908-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:58:35,916-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:58:40,067-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:58:45,924-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:58:50,040-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:58:55,931-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:59:05,939-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:59:15,947-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-06 21:59:17,385-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-06 21:59:25,955-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:59:26,514-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:59:35,963-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 21:59:45,970-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 21:59:54,703-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 21:59:55,978-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 22:00:02,989-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 22:00:05,986-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:00:15,994-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:00:26,002-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:00:32,021-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:00:36,010-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:00:39,462-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:00:46,018-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:00:56,026-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:01:06,034-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 22:01:09,339-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 22:01:15,936-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 22:01:16,042-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:01:26,050-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:01:36,058-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:01:46,066-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:01:46,657-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:01:52,410-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:01:56,074-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:02:06,082-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:02:16,090-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 22:02:23,975-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 22:02:26,098-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:02:28,884-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:02:36,106-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:02:46,114-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:02:56,122-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:03:01,293-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:03:05,358-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:03:06,130-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:03:16,138-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:03:26,146-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:03:36,154-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:03:38,612-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:03:41,832-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:03:46,162-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:03:56,170-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:04:06,178-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:04:15,930-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:04:16,186-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:04:18,306-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:04:26,194-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:04:36,202-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:04:46,210-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:04:53,248-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:04:54,780-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:04:56,220-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:05:06,229-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:05:16,237-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:05:26,245-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 22:05:30,566-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:05:31,255-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:05:36,253-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:05:46,261-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:05:56,268-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:06:06,277-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:06:07,729-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:06:07,884-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:06:16,284-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:06:26,292-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:06:36,300-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 22:06:44,204-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:06:45,202-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:06:46,308-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:06:56,316-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:07:06,324-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:07:16,332-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:07:20,677-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:07:22,520-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:07:26,340-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:07:36,348-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:07:46,356-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:07:56,364-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 4ms
-2013-11-06 22:07:57,154-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 4ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:07:59,838-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:08:06,372-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:08:16,380-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:08:26,388-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:08:33,625-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:08:36,396-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 22:08:37,156-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 22:08:46,404-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:08:56,412-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:09:06,420-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 22:09:10,099-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:09:14,474-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:09:16,428-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:09:26,436-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:09:36,444-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:09:46,452-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:09:46,573-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-06 22:09:51,792-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-06 22:09:56,460-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:10:06,467-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:10:16,475-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 22:10:23,048-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 22:10:26,483-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:10:29,111-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:10:36,491-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:10:46,499-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:10:56,507-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:10:59,521-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:11:06,429-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:11:06,515-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:11:16,522-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:11:26,530-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 22:11:35,995-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 22:11:36,538-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 22:11:43,747-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 22:11:46,546-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:11:56,554-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:12:06,562-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:12:12,469-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:12:16,570-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:12:21,065-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:12:26,579-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:12:36,587-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:12:46,595-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 22:12:48,943-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 22:12:56,603-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 22:12:58,383-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 22:13:06,611-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:13:16,619-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:13:25,417-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:13:26,627-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:13:35,701-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:13:36,635-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:13:46,643-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:13:56,651-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 22:14:01,891-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 22:14:06,659-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 22:14:13,019-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 22:14:16,667-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:14:26,675-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:14:36,682-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:14:38,365-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:14:46,690-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:14:50,337-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:14:56,698-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:15:06,706-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 22:15:14,840-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 22:15:16,714-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:15:26,722-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:15:27,655-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:15:36,730-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:15:46,738-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:15:51,314-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:15:56,746-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 22:16:04,973-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 22:16:06,754-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:16:16,762-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:16:26,770-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:16:27,788-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:16:36,778-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:16:42,291-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:16:46,786-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:16:56,794-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 22:17:04,263-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 22:17:06,802-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:17:16,810-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 22:17:19,609-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 22:17:26,818-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:17:36,826-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:17:40,736-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:17:46,834-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:17:56,842-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:17:56,927-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:18:06,850-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:18:16,858-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 22:18:17,211-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 22:18:26,866-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-06 22:18:34,245-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-06 22:18:36,874-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:18:46,882-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:18:53,684-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:18:56,890-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:19:06,898-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:19:11,564-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:19:16,906-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:19:26,914-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:19:30,158-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:19:36,922-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:19:46,930-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-06 22:19:48,881-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-06 22:19:56,938-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:20:06,632-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:20:06,946-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:20:16,954-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 22:20:26,200-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 22:20:26,962-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:20:36,970-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:20:43,106-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:20:46,978-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:20:56,986-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:21:03,518-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:21:06,994-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:21:17,002-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:21:19,580-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:21:27,009-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:21:37,017-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 22:21:40,836-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 22:21:47,025-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 22:21:56,055-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 22:21:57,033-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:22:07,041-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:22:17,048-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:22:18,154-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:22:27,056-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 22:22:32,529-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 22:22:37,064-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:22:47,072-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-06 22:22:55,472-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-06 22:22:57,080-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:23:07,088-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 22:23:09,003-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 22:23:17,096-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:23:27,104-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:23:32,791-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:23:37,112-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:23:45,476-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:23:47,120-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:23:57,128-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:24:07,136-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:24:10,109-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:24:17,144-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 22:24:21,951-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 22:24:27,152-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:24:37,161-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:24:47,169-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 22:24:47,427-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 22:24:57,177-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:24:58,424-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:25:07,185-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:25:17,193-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:25:24,745-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:25:27,201-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 22:25:34,899-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 22:25:37,210-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:25:47,218-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:25:57,226-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-06 22:26:02,063-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-06 22:26:07,234-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:26:11,372-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:26:17,242-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:26:27,250-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:26:37,258-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 22:26:39,382-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 22:26:47,267-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:26:47,846-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:26:57,275-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:27:07,283-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:27:16,700-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:27:17,291-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:27:24,320-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:27:27,299-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:27:37,308-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:27:47,316-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 22:27:54,018-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 22:27:57,324-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 22:28:00,795-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 22:28:07,332-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:28:17,340-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:28:27,349-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:28:31,336-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:28:37,268-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:28:37,357-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:28:47,365-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:28:57,373-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:29:07,381-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-06 22:29:08,653-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:29:13,742-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:29:17,389-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:29:27,398-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:29:37,406-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 22:29:45,972-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 22:29:47,414-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:29:50,216-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:29:57,422-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:30:07,430-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:30:17,438-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:30:23,290-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 22:30:26,691-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 22:30:27,447-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:30:37,455-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:30:47,463-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:30:57,471-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 22:31:00,608-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:31:03,164-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:31:07,479-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:31:17,487-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:31:27,495-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:31:37,503-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:31:37,926-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:31:39,638-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:31:47,511-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:31:57,519-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:32:07,527-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:32:15,244-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:32:16,112-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:32:17,535-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:32:27,544-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:32:37,552-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:32:47,560-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 22:32:52,562-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:32:52,586-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:32:57,568-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:33:07,576-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:33:17,585-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:33:27,593-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 22:33:29,060-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:33:29,880-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:33:37,601-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:33:47,610-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:33:57,618-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:34:05,534-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-06 22:34:07,198-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-06 22:34:07,626-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:34:17,634-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:34:27,643-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:34:37,651-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:34:42,008-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:34:44,517-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:34:47,659-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:34:57,667-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:35:07,676-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:35:17,684-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 22:35:18,483-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:35:21,835-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:35:27,692-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:35:37,700-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:35:47,709-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:35:54,956-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:35:57,717-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 22:35:59,153-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 22:36:07,725-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:36:17,733-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:36:27,741-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 22:36:31,431-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:36:36,471-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:36:37,749-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:36:47,757-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:36:57,765-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:37:07,773-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:37:07,904-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 22:37:13,789-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 22:37:17,780-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:37:27,788-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:37:37,796-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:37:44,378-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:37:47,804-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:37:51,107-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:37:57,812-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:38:07,820-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:38:17,828-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:38:20,852-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:38:27,836-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-06 22:38:28,425-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-06 22:38:37,843-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:38:47,851-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:38:57,326-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:38:57,859-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:39:05,744-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:39:07,868-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:39:17,876-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:39:27,884-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 22:39:33,801-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 22:39:37,891-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:39:43,062-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:39:47,900-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:39:57,908-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:40:07,916-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:40:10,274-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:40:17,924-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 22:40:20,380-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 22:40:27,932-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:40:37,940-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 22:40:46,749-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 22:40:47,948-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:40:57,698-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:40:57,956-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:41:07,965-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:41:17,973-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:41:23,222-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:41:27,981-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 22:41:35,016-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 22:41:37,988-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:41:47,996-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:41:58,004-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:41:59,696-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:42:08,012-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:42:12,334-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:42:18,020-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:42:28,027-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:42:36,170-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:42:38,035-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:42:48,043-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 22:42:49,652-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 22:42:58,051-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:43:08,059-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 22:43:12,645-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 22:43:18,067-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:43:26,970-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:43:28,074-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:43:38,082-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:43:48,090-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:43:49,118-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:43:58,098-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-06 22:44:04,287-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-06 22:44:08,105-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:44:18,113-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:44:25,592-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:44:28,121-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:44:38,129-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:44:41,606-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:44:48,136-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:44:58,144-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:45:02,066-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:45:08,152-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:45:18,159-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:45:18,924-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:45:28,167-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:45:38,175-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:45:38,540-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:45:48,183-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 22:45:56,242-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 22:45:58,190-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:46:08,198-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 22:46:15,014-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 22:46:18,206-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:46:28,214-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:46:33,560-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:46:38,222-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:46:48,230-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:46:51,488-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:46:58,238-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:47:08,245-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 22:47:10,878-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 22:47:18,253-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 7ms
-2013-11-06 22:47:27,968-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 7ms
-No streams
-2013-11-06 22:47:28,261-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:47:38,269-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:47:48,196-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:47:48,277-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:47:58,285-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:48:04,436-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:48:08,293-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:48:18,301-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-06 22:48:25,514-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-06 22:48:28,309-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:48:38,317-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:48:40,910-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:48:48,324-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:48:58,332-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:49:02,832-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:49:08,340-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 22:49:17,385-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 22:49:18,348-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:49:28,356-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:49:38,364-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:49:40,150-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:49:48,372-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 22:49:53,859-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 22:49:58,379-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:50:08,387-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:50:17,468-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:50:18,395-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:50:28,402-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:50:30,332-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:50:38,410-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:50:48,418-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:50:54,786-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:50:58,426-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 22:51:06,807-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 22:51:08,434-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:51:18,442-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:51:28,450-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:51:32,104-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:51:38,458-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:51:43,280-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:51:48,466-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:51:58,474-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:52:08,481-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:52:09,422-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:52:18,490-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:52:19,754-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:52:28,497-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:52:38,505-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 22:52:46,740-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 22:52:48,513-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:52:56,228-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:52:58,521-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:53:08,529-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:53:18,537-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:53:24,058-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:53:28,545-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 22:53:32,703-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 22:53:38,553-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:53:48,561-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:53:58,569-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:54:01,376-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:54:08,577-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:54:09,176-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:54:18,585-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:54:28,593-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:54:38,601-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:54:38,694-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:54:45,650-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:54:48,609-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:54:58,617-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:55:08,625-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:55:16,012-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:55:18,633-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:55:22,124-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:55:28,641-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:55:38,649-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:55:48,657-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:55:53,330-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 22:55:58,599-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 22:55:58,665-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:56:08,673-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:56:18,681-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:56:28,689-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:56:30,648-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:56:35,072-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:56:38,697-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:56:48,705-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:56:58,713-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:57:07,966-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:57:08,721-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 22:57:11,547-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 22:57:18,729-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:57:28,737-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:57:38,745-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:57:45,284-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:57:48,020-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:57:48,753-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:57:58,761-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:58:08,769-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:58:18,777-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:58:22,602-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:58:24,494-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:58:28,785-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:58:38,793-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:58:48,801-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:58:58,809-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-06 22:58:59,920-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:59:00,968-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:59:08,817-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:59:18,825-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:59:28,833-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:59:37,239-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 22:59:37,442-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 22:59:38,841-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:59:48,849-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 22:59:58,857-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:00:08,865-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 23:00:13,917-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:00:14,557-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:00:18,873-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:00:28,881-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:00:38,889-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:00:48,897-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:00:50,390-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:00:51,875-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:00:58,905-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:01:08,913-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:01:18,921-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:01:26,864-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:01:28,930-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:01:29,193-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:01:38,938-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:01:48,946-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:01:58,954-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:02:03,338-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 23:02:06,511-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 23:02:08,962-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:02:18,971-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:02:28,979-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:02:38,987-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:02:39,813-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:02:43,830-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:02:48,995-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:02:59,003-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:03:09,012-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:03:16,287-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:03:19,020-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:03:21,148-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:03:29,028-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:03:39,036-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:03:49,044-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:03:52,762-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:03:58,466-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:03:59,052-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:04:09,060-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:04:19,068-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:04:29,076-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:04:29,236-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:04:35,784-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:04:39,084-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:04:49,092-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:04:59,100-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 23:05:05,711-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 23:05:09,108-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-06 23:05:13,102-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-06 23:05:19,116-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:05:29,124-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:05:39,133-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 23:05:42,185-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 23:05:49,141-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:05:50,420-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:05:59,149-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:06:09,157-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:06:18,659-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:06:19,165-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:06:27,738-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:06:29,173-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:06:39,181-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:06:49,189-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:06:55,133-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:06:59,198-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:07:05,056-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:07:09,206-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:07:19,214-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:07:29,222-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 23:07:31,607-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 23:07:39,231-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:07:42,374-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:07:49,239-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:07:59,247-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 23:08:08,082-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 23:08:09,255-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:08:19,264-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:08:19,692-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:08:29,272-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:08:39,280-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 23:08:44,556-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 23:08:49,289-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:08:57,010-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:08:59,297-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:09:09,305-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:09:19,313-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:09:21,029-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:09:29,321-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:09:34,328-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:09:39,330-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:09:49,338-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 23:09:57,504-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 23:09:59,346-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:10:09,354-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:10:11,646-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:10:19,362-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:10:29,370-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 23:10:33,978-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 23:10:39,378-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:10:48,964-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:10:49,386-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:10:59,394-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:11:09,402-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 23:11:10,452-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 23:11:19,409-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:11:26,282-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:11:29,417-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:11:39,425-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:11:46,925-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:11:49,433-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:11:59,441-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:12:03,600-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:12:09,448-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:12:19,456-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:12:23,399-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:12:29,464-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:12:39,472-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 23:12:40,918-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 23:12:49,480-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:12:59,487-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:12:59,873-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:13:09,495-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:13:18,237-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:13:19,503-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:13:29,511-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:13:36,347-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:13:39,519-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:13:49,527-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 23:13:55,554-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 23:13:59,535-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:14:09,543-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:14:12,821-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:14:19,551-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:14:29,558-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:14:32,873-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:14:39,566-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 23:14:49,296-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 23:14:49,574-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:14:59,582-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:15:09,590-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:15:10,191-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:15:19,598-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:15:25,769-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:15:29,606-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:15:39,614-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:15:47,509-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:15:49,622-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:15:59,630-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:16:02,243-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:16:09,637-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:16:19,645-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:16:24,827-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:16:29,653-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:16:38,717-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:16:39,661-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:16:49,669-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:16:59,677-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:17:02,145-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:17:09,685-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:17:15,191-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:17:19,693-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:17:29,701-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:17:39,463-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:17:39,709-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:17:49,716-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 23:17:51,665-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 23:17:59,724-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:18:09,732-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 23:18:16,781-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 23:18:19,740-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:18:28,139-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:18:29,748-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:18:39,756-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:18:49,764-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:18:54,100-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:18:59,772-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 23:19:04,614-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 23:19:09,780-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:19:19,788-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:19:29,795-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:19:31,418-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:19:39,803-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:19:41,087-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:19:49,811-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:19:59,819-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:20:08,736-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:20:09,827-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:20:17,561-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:20:19,835-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:20:29,842-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:20:39,850-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:20:46,054-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:20:49,858-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:20:54,035-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:20:59,866-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:21:09,874-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:21:19,882-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:21:23,372-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:21:29,890-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 23:21:30,510-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 23:21:39,898-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:21:49,905-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:21:59,913-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:22:00,690-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 23:22:06,983-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 23:22:09,921-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:22:19,929-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:22:29,937-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 23:22:38,007-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 23:22:39,945-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 23:22:43,458-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 23:22:49,953-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:22:59,961-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:23:09,969-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:23:15,326-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:23:19,931-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:23:19,977-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:23:29,985-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:23:39,993-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:23:50,001-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 23:23:52,644-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:23:56,405-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:24:00,009-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:24:10,017-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:24:20,025-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:24:29,962-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:24:30,033-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 23:24:32,879-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 23:24:40,041-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:24:50,049-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:25:00,057-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:25:07,280-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:25:09,353-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:25:10,065-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:25:20,073-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:25:30,081-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:25:40,089-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 23:25:44,599-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 23:25:45,827-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 23:25:50,097-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:26:00,105-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:26:10,113-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:26:20,122-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:26:21,916-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:26:22,301-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:26:30,130-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:26:40,138-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:26:50,146-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 23:26:58,775-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:26:59,234-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:27:00,154-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:27:10,162-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:27:20,170-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:27:30,179-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 23:27:35,250-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:27:36,552-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:27:40,187-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:27:50,195-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:28:00,203-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:28:10,211-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 23:28:11,723-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:28:13,870-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:28:20,219-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:28:30,227-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:28:40,235-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 23:28:48,198-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 23:28:50,243-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 23:28:51,187-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 23:29:00,251-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:29:10,259-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:29:20,267-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 23:29:24,672-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:29:28,506-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:29:30,275-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:29:40,283-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:29:50,291-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:30:00,299-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:30:01,146-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 23:30:05,823-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 23:30:10,307-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:30:20,315-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:30:30,323-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 23:30:37,621-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 23:30:40,331-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:30:43,142-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:30:50,339-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:31:00,347-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:31:10,355-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:31:14,094-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:31:20,363-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:31:20,460-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:31:30,371-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:31:40,379-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:31:50,387-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 23:31:50,569-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:31:57,778-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:32:00,395-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:32:10,403-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:32:20,411-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 23:32:27,042-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 23:32:30,419-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:32:35,096-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:32:40,427-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:32:50,436-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:33:00,444-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 23:33:03,517-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 23:33:10,452-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:33:12,414-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:33:20,460-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:33:30,468-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 23:33:39,990-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 23:33:40,477-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:33:49,732-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:33:50,485-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:34:00,493-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:34:10,502-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 23:34:16,465-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 23:34:20,510-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:34:27,050-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:34:30,518-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:34:40,526-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:34:50,534-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 23:34:52,940-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 23:35:00,542-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 23:35:04,368-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 23:35:10,550-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:35:20,558-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 23:35:29,414-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 23:35:30,567-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:35:40,575-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 23:35:41,686-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 23:35:50,584-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:36:00,591-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 23:36:05,888-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 23:36:10,600-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:36:19,005-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:36:20,609-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:36:30,617-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:36:40,626-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 23:36:42,362-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 23:36:50,635-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 23:36:56,324-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 23:37:00,643-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:37:10,651-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-06 23:37:18,837-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-06 23:37:20,660-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:37:30,668-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:37:33,641-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:37:40,676-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:37:50,684-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:37:55,310-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:38:00,693-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:38:10,701-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 23:38:10,959-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 23:38:20,709-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:38:30,719-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 23:38:31,785-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 23:38:40,727-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 23:38:48,278-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 23:38:50,735-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:39:00,744-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:39:08,258-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:39:10,752-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:39:20,760-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:39:25,595-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:39:30,768-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:39:40,776-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 23:39:44,733-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 23:39:50,784-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:40:00,792-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 23:40:02,913-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 23:40:10,800-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:40:20,808-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:40:21,207-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:40:30,816-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 23:40:40,232-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 23:40:40,824-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:40:50,831-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:40:57,681-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:41:00,839-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:41:10,847-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:41:17,549-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:41:20,854-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:41:30,862-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 23:41:34,156-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 23:41:40,870-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:41:50,878-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 23:41:54,867-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 23:42:00,885-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 23:42:10,630-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 23:42:10,893-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:42:20,901-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:42:30,909-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:42:32,186-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:42:40,917-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:42:47,103-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:42:50,924-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:43:00,933-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 23:43:09,504-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 23:43:10,941-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:43:20,949-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:43:23,577-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:43:30,957-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:43:40,965-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:43:46,822-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:43:50,973-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:44:00,051-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:44:00,982-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:44:10,990-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:44:20,998-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:44:24,140-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:44:31,006-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 23:44:36,526-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 23:44:41,013-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:44:51,021-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:45:01,029-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:45:01,458-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:45:11,037-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 23:45:12,999-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 23:45:21,045-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:45:31,053-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:45:38,776-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:45:41,061-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 23:45:49,474-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 23:45:51,069-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:46:01,077-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:46:11,085-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-06 23:46:16,094-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-06 23:46:21,093-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:46:25,947-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:46:31,101-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:46:41,109-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:46:51,116-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:46:53,413-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:47:01,124-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 23:47:02,422-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 23:47:11,132-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:47:21,140-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:47:30,731-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:47:31,148-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 23:47:38,895-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 23:47:41,156-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:47:51,164-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:48:01,172-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:48:08,049-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:48:11,180-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 23:48:15,370-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 23:48:21,188-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:48:31,196-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:48:41,203-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:48:45,367-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:48:51,211-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 23:48:51,843-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 23:49:01,220-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:49:11,227-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:49:21,235-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:49:22,685-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:49:28,317-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:49:31,244-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:49:41,252-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:49:51,260-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:50:00,003-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:50:01,268-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 23:50:04,791-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 23:50:11,276-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:50:21,284-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:50:31,292-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:50:37,321-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-2013-11-06 23:50:41,267-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-No streams
-2013-11-06 23:50:41,300-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:50:51,308-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:51:01,316-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:51:11,324-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:51:14,639-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 23:51:17,739-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-06 23:51:21,332-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:51:31,339-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:51:41,347-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:51:51,355-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:51:51,957-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:51:54,213-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:52:01,363-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:52:11,371-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:52:21,379-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-06 23:52:29,275-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:52:30,687-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:52:31,386-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:52:41,394-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:52:51,402-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:53:01,410-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:53:06,594-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 23:53:07,162-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-06 23:53:11,418-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:53:21,426-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:53:31,434-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:53:41,442-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 23:53:43,635-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:53:43,912-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:53:51,450-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:54:01,458-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:54:11,466-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 23:54:20,110-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-06 23:54:21,229-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-06 23:54:21,474-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:54:31,482-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:54:41,490-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:54:51,498-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:54:56,583-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:54:58,548-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:55:01,506-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:55:11,514-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:55:21,522-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:55:31,530-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 23:55:33,058-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:55:35,866-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:55:41,538-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:55:51,546-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:56:01,554-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:56:09,531-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:56:11,562-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-06 23:56:13,184-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-06 23:56:21,571-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:56:31,579-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:56:41,587-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-06 23:56:46,006-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:56:50,503-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:56:51,595-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:57:01,603-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:57:11,611-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:57:21,619-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:57:22,479-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:57:27,821-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:57:31,627-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:57:41,635-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:57:51,643-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:57:58,953-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:58:01,651-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:58:05,139-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:58:11,659-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:58:21,667-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:58:31,675-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:58:35,427-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:58:41,683-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:58:42,457-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:58:51,690-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:59:01,698-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:59:11,706-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-06 23:59:11,902-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-06 23:59:19,774-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-06 23:59:21,714-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:59:31,721-0600 INFO Multiplexer No streams
-No streams
-2013-11-06 23:59:41,729-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:59:48,376-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-06 23:59:51,736-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-06 23:59:57,093-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:00:01,744-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:00:11,752-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:00:21,760-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 00:00:24,851-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 00:00:31,768-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:00:34,411-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:00:41,776-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:00:51,784-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:01:01,324-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:01:01,792-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:01:11,729-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:01:11,800-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:01:21,807-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:01:31,815-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:01:37,799-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:01:41,825-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:01:49,048-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:01:51,835-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:02:01,848-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:02:11,860-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:02:14,273-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:02:21,870-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:02:26,365-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:02:31,878-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:02:41,888-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:02:50,748-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:02:51,898-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:03:01,918-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:03:03,683-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:03:11,927-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:03:21,934-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 00:03:27,223-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 00:03:31,944-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:03:41,001-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:03:41,957-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:03:51,967-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:04:01,979-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:04:03,696-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:04:11,988-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 00:04:18,320-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 00:04:21,999-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:04:32,012-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 00:04:40,171-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 00:04:42,021-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:04:52,032-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:04:55,638-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:05:02,042-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:05:12,052-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:05:16,644-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:05:22,061-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:05:32,070-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 00:05:32,956-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 00:05:42,080-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:05:52,089-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:05:53,118-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:06:02,098-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:06:10,274-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:06:12,107-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:06:22,115-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:06:29,592-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 00:06:32,125-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:06:42,134-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 00:06:47,592-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 00:06:52,143-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:07:02,152-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 00:07:06,067-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 00:07:12,162-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:07:22,170-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:07:24,909-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:07:32,178-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:07:42,186-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:07:42,540-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:07:52,195-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:08:02,203-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:08:02,228-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 00:08:12,211-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:08:19,014-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 00:08:22,219-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:08:32,227-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:08:39,545-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:08:42,235-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:08:52,243-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:08:55,488-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:09:02,251-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:09:12,259-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:09:16,863-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 00:09:22,267-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:09:31,963-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 00:09:32,275-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:09:42,282-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:09:52,291-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:09:54,181-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:10:02,298-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:10:08,437-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:10:12,306-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:10:22,314-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:10:31,499-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:10:32,322-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:10:42,330-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:10:44,911-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:10:52,338-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:11:02,346-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:11:08,817-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:11:12,354-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:11:21,385-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:11:22,362-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:11:32,370-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:11:42,378-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:11:46,135-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:11:52,386-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 00:11:57,860-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 00:12:02,393-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:12:12,401-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:12:22,409-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 00:12:23,453-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 00:12:32,417-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:12:34,333-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 00:12:42,425-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:12:52,433-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:13:00,772-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:13:02,441-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 00:13:10,808-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 00:13:12,449-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:13:22,457-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:13:32,465-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:13:38,090-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 00:13:42,473-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:13:47,281-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:13:52,480-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:14:02,488-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:14:12,496-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:14:15,408-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:14:22,504-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 00:14:23,756-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 00:14:32,512-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:14:42,520-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:14:52,528-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:14:52,726-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:15:00,229-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 00:15:02,535-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:15:12,543-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:15:22,551-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 00:15:30,045-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 00:15:32,559-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 20ms
-2013-11-07 00:15:36,722-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 20ms
-No streams
-2013-11-07 00:15:42,567-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:15:52,575-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:16:02,583-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:16:07,362-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:16:12,591-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:16:13,177-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:16:22,599-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:16:32,607-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:16:42,614-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:16:44,680-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:16:49,651-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:16:52,622-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:17:02,630-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:17:12,638-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:17:21,998-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:17:22,646-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:17:26,125-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:17:32,654-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:17:42,662-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:17:52,670-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:17:59,316-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 00:18:02,600-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 00:18:02,678-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:18:12,686-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:18:22,694-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:18:32,701-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 00:18:36,635-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:18:39,073-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:18:42,709-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:18:52,717-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:19:02,725-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:19:12,733-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:19:13,952-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:19:15,547-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:19:22,741-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:19:32,749-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:19:42,757-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:19:51,270-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:19:52,021-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:19:52,765-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:20:02,773-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:20:12,781-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:20:22,789-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 00:20:28,496-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:20:28,588-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:20:32,797-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:20:42,805-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:20:52,813-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:21:02,821-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:21:04,969-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:21:05,906-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 00:21:12,828-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:21:22,836-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:21:32,844-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:21:41,443-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:21:42,852-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 00:21:43,223-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 00:21:52,860-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:22:02,868-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:22:12,876-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 5ms
-2013-11-07 00:22:17,921-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 5ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:22:20,542-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 00:22:22,884-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:22:32,892-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:22:42,900-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:22:52,908-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:22:54,391-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:22:57,860-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 00:23:02,916-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:23:12,923-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:23:22,931-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:23:30,865-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 00:23:32,939-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:23:35,178-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:23:42,947-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:23:52,955-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:24:02,963-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:24:07,339-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:24:12,496-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 00:24:12,971-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:24:22,979-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:24:32,987-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:24:42,995-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 00:24:43,814-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:24:49,814-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:24:53,003-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:25:03,011-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:25:13,018-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:25:20,287-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:25:23,026-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:25:27,132-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 00:25:33,034-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:25:43,042-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:25:53,050-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 00:25:56,762-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 00:26:03,058-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:26:04,450-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:26:13,066-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:26:23,074-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:26:33,082-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:26:33,235-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:26:41,768-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 00:26:43,090-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:26:53,099-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:27:03,107-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:27:09,709-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:27:13,115-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:27:19,086-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:27:23,123-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:27:33,130-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:27:43,139-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 00:27:46,182-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 00:27:53,147-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:27:56,404-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 00:28:03,155-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:28:13,163-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:28:22,657-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:28:23,170-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:28:33,178-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 00:28:33,722-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 00:28:43,186-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:28:53,194-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 00:28:59,133-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 00:29:03,202-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:29:11,041-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 00:29:13,210-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:29:23,218-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:29:33,226-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:29:35,606-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:29:43,234-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 00:29:48,358-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 00:29:53,242-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:30:03,250-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 00:30:12,081-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 00:30:13,258-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:30:23,266-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:30:25,677-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 00:30:33,274-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:30:43,282-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 00:30:48,555-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 00:30:53,290-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:31:02,995-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 00:31:03,298-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:31:13,305-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:31:23,313-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:31:25,028-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:31:33,321-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:31:40,313-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 00:31:43,329-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:31:53,337-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:32:01,502-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:32:03,344-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:32:13,352-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:32:17,631-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 00:32:23,360-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:32:33,368-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 00:32:37,977-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 00:32:43,376-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:32:53,384-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:32:54,949-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 00:33:03,391-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:33:13,399-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:33:14,450-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 00:33:23,407-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:33:32,267-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 00:33:33,415-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:33:43,423-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:33:50,924-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:33:53,431-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:34:03,439-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:34:09,585-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:34:13,446-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:34:23,454-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 00:34:27,399-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 00:34:33,462-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:34:43,470-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:34:46,903-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 00:34:53,477-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:35:03,485-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:35:03,872-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:35:13,493-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:35:23,501-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:35:24,221-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:35:33,508-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:35:40,346-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:35:43,516-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:35:53,524-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:36:01,539-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 00:36:03,531-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:36:13,539-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:36:16,820-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 00:36:23,547-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:36:33,554-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:36:38,857-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:36:43,562-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:36:53,294-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:36:53,570-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:37:03,578-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:37:13,586-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:37:16,175-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 00:37:23,594-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:37:29,768-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:37:33,602-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:37:43,610-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:37:53,494-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 00:37:53,621-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:38:03,629-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:38:06,242-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:38:13,637-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:38:23,645-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:38:30,812-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 00:38:33,654-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 00:38:42,718-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 00:38:43,662-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:38:53,670-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:39:03,678-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:39:08,130-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 00:39:13,686-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:39:19,191-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:39:23,694-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:39:33,702-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:39:43,710-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:39:45,448-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 00:39:53,718-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:39:55,665-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:40:03,727-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:40:13,735-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:40:22,766-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 00:40:23,743-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:40:32,139-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:40:33,751-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:40:43,759-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:40:53,767-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:41:00,084-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 00:41:03,775-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 00:41:08,614-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 00:41:13,783-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:41:23,791-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:41:33,798-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:41:37,402-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 00:41:43,806-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:41:45,087-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:41:53,814-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:42:03,822-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:42:13,830-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:42:14,720-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 00:42:21,562-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 00:42:23,837-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:42:33,845-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:42:43,853-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:42:52,038-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 00:42:53,861-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:42:58,035-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:43:03,869-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:43:13,877-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:43:23,884-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 00:43:29,355-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 00:43:33,892-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:43:34,509-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:43:43,900-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:43:53,908-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:44:03,916-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:44:06,674-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:44:10,983-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 00:44:13,924-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:44:23,932-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:44:33,940-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:44:43,948-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:44:43,992-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 28ms
-2013-11-07 00:44:47,484-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 28ms
-No streams
-2013-11-07 00:44:53,956-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:45:03,964-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:45:13,972-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 00:45:21,311-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:45:23,931-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 00:45:23,980-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:45:33,988-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:45:43,996-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:45:54,004-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:45:58,628-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:46:00,405-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:46:04,012-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:46:14,020-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:46:24,028-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:46:34,036-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:46:35,946-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:46:36,879-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 00:46:44,044-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:46:54,052-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:47:04,060-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:47:13,264-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 00:47:13,354-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 00:47:14,068-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:47:24,076-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:47:34,084-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:47:44,091-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:47:49,827-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:47:50,582-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 00:47:54,100-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:48:04,108-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:48:14,115-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:48:24,123-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:48:26,301-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:48:27,900-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 00:48:34,131-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:48:44,139-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:48:54,147-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:49:02,775-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 00:49:04,155-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:49:05,218-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 00:49:14,163-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:49:24,171-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:49:34,179-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:49:39,249-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:49:42,536-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 00:49:44,187-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:49:54,195-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:50:04,203-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:50:14,211-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:50:15,723-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:50:19,854-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 00:50:24,219-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:50:34,227-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:50:44,235-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:50:52,197-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 00:50:54,243-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:50:57,172-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 00:51:04,251-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:51:14,259-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:51:24,267-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:51:28,671-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:51:34,275-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:51:34,490-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 00:51:44,283-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:51:54,291-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:52:04,299-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:52:05,145-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:52:11,808-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:52:14,307-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:52:24,315-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:52:34,323-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:52:41,619-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:52:44,331-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:52:49,126-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 00:52:54,339-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:53:04,347-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:53:14,355-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:53:18,093-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 00:53:24,363-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:53:26,444-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 00:53:34,371-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:53:44,379-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:53:54,387-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:53:54,567-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:54:03,762-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 00:54:04,395-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:54:14,403-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:54:24,411-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:54:31,041-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:54:34,419-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:54:41,080-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 00:54:44,427-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:54:54,435-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:55:04,444-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 00:55:07,516-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 00:55:14,452-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:55:18,398-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:55:24,460-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:55:34,468-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:55:43,990-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 00:55:44,476-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:55:54,484-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:55:55,716-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 00:56:04,492-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:56:14,499-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 00:56:20,464-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 00:56:24,507-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:56:33,034-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 00:56:34,515-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:56:44,522-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:56:54,530-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 00:56:56,939-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 00:57:04,538-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:57:10,352-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 00:57:14,546-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:57:24,555-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 00:57:33,413-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 00:57:34,562-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:57:44,571-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 00:57:47,671-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 00:57:54,579-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:58:04,587-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 00:58:09,887-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 00:58:14,595-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:58:24,604-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:58:24,988-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 00:58:34,612-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:58:44,621-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 00:58:46,361-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 00:58:54,629-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 00:59:02,306-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 00:59:04,638-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:59:14,649-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 00:59:22,835-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 00:59:24,660-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:59:34,669-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 00:59:39,625-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 00:59:44,680-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 00:59:54,690-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 00:59:59,309-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 01:00:04,701-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:00:14,711-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 01:00:16,943-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 01:00:24,721-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:00:34,731-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 01:00:35,783-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 01:00:44,750-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 01:00:54,260-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 01:00:54,769-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:01:04,771-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 01:01:12,257-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 01:01:14,772-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:01:24,774-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 01:01:31,579-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 01:01:34,775-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:01:44,779-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 01:01:48,731-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 01:01:54,796-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:02:04,804-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:02:08,896-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:02:14,813-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:02:24,821-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:02:25,204-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:02:34,832-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:02:44,843-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 01:02:46,215-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 01:02:54,853-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 01:03:01,679-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 01:03:04,862-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:03:14,871-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:03:23,532-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:03:24,881-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:03:34,891-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:03:38,152-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:03:44,899-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:03:54,909-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 01:04:00,850-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 01:04:04,917-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 01:04:14,627-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 01:04:14,927-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:04:24,938-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:04:34,951-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 01:04:38,168-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 01:04:44,964-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 01:04:51,101-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 01:04:54,973-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:05:04,984-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:05:14,992-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 01:05:15,486-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 01:05:25,002-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:05:27,574-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:05:35,012-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:05:45,024-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:05:52,804-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:05:55,035-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 01:06:04,049-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 01:06:05,046-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:06:15,054-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:06:25,063-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 01:06:30,123-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 01:06:35,070-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 01:06:40,522-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 01:06:45,078-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:06:55,086-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:07:05,094-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 01:07:07,442-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 01:07:15,101-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 4ms
-2013-11-07 01:07:16,999-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 4ms
-No streams
-2013-11-07 01:07:25,109-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:07:35,117-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 01:07:44,761-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 01:07:45,125-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 01:07:53,471-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 01:07:55,133-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:08:05,141-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:08:15,149-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 01:08:22,079-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 01:08:25,157-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:08:29,944-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:08:35,167-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:08:45,175-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:08:55,183-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 01:08:59,396-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 01:09:05,191-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 01:09:06,419-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 01:09:15,200-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:09:25,209-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:09:35,220-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 01:09:36,714-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 01:09:42,893-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 01:09:45,230-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:09:55,239-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:10:05,250-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:10:14,032-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:10:15,260-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:10:19,366-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:10:25,272-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:10:35,282-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:10:45,291-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 01:10:51,351-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 01:10:55,302-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 01:10:55,840-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 01:11:05,310-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:11:15,321-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:11:25,330-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 01:11:28,669-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 01:11:32,315-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 01:11:35,341-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:11:45,351-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:11:55,361-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:12:05,372-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:12:05,987-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:12:08,788-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:12:15,381-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:12:25,391-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:12:35,401-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 01:12:43,306-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:12:45,262-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:12:45,410-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:12:55,419-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:13:05,429-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:13:15,441-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 01:13:20,624-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:13:21,736-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:13:25,451-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:13:35,460-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:13:45,469-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:13:55,477-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 01:13:57,942-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 01:13:58,211-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 01:14:05,486-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:14:15,494-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:14:25,502-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:14:34,684-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:14:35,260-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:14:35,510-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:14:45,518-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:14:55,526-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:15:05,535-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:15:11,159-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:15:12,577-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:15:15,544-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:15:25,553-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:15:35,561-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:15:45,569-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:15:47,634-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:15:49,895-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:15:55,577-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:16:05,585-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:16:15,593-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 01:16:24,109-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 01:16:25,601-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:16:27,213-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:16:35,610-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:16:45,619-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:16:55,628-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:17:00,582-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:17:04,531-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:17:05,636-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:17:15,645-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:17:25,656-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:17:35,665-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 01:17:37,057-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 01:17:41,850-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 01:17:45,673-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:17:55,682-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:18:05,691-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:18:13,530-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:18:15,703-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:18:19,167-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:18:25,714-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:18:35,724-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:18:45,732-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 01:18:50,005-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 01:18:55,740-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 01:18:56,486-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 01:19:05,748-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:19:15,756-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:19:25,764-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 01:19:26,479-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 01:19:33,804-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 01:19:35,772-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:19:45,779-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:19:55,787-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:20:02,952-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:20:05,795-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 01:20:11,121-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 01:20:15,802-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:20:25,810-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:20:35,818-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:20:39,426-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:20:45,825-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 01:20:48,440-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 01:20:55,833-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:21:05,840-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:21:15,848-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:21:15,900-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:21:25,757-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:21:25,856-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:21:35,863-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:21:45,871-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 01:21:52,375-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 01:21:55,880-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 01:22:03,076-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 01:22:05,890-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:22:15,900-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:22:25,919-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 01:22:28,849-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 01:22:35,920-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:22:40,393-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:22:45,924-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:22:55,926-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:23:05,322-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:23:05,941-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:23:15,950-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 01:23:17,711-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 01:23:25,959-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:23:35,969-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:23:41,796-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:23:45,978-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:23:55,029-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:23:55,987-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:24:05,998-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:24:16,011-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 01:24:18,271-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 01:24:26,025-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 01:24:32,348-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 01:24:36,035-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:24:46,045-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 01:24:54,745-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 01:24:56,049-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:25:06,061-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:25:09,665-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:25:16,077-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:25:26,086-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:25:31,218-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:25:36,095-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:25:46,104-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 01:25:46,984-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 01:25:56,112-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:26:06,120-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 01:26:07,693-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 01:26:16,129-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 01:26:24,302-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 01:26:26,137-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:26:36,145-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 01:26:44,167-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 01:26:46,147-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:26:56,158-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:27:01,619-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:27:06,170-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:27:16,186-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 01:27:20,641-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 01:27:26,195-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:27:36,215-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:27:38,937-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:27:46,227-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:27:56,245-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 01:27:57,115-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 01:28:06,255-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:28:16,255-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:28:16,262-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:28:26,272-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 01:28:33,589-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 01:28:36,282-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:28:46,294-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 01:28:53,574-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 01:28:56,305-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:29:06,316-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:29:10,062-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:29:16,325-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:29:26,335-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:29:30,891-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:29:36,344-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:29:46,353-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:29:46,536-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:29:56,363-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:30:06,372-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:30:08,209-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:30:16,383-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 01:30:23,011-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 01:30:26,392-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:30:36,409-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:30:45,529-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:30:46,428-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:30:56,443-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:30:59,484-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:31:06,450-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:31:16,467-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:31:22,847-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:31:26,487-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:31:35,959-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:31:36,507-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:31:46,521-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:31:56,529-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:32:00,165-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:32:06,537-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 01:32:12,434-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 01:32:16,546-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:32:26,555-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:32:36,563-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:32:37,483-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:32:46,571-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:32:48,907-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:32:56,578-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:33:06,586-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:33:14,801-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:33:16,594-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:33:25,381-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:33:26,601-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:33:36,610-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:33:46,617-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:33:52,119-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:33:56,625-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:34:01,855-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:34:06,634-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:34:16,644-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:34:26,654-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 01:34:29,438-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 01:34:36,664-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:34:38,329-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:34:46,684-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:34:56,695-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:35:06,705-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 01:35:06,756-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:35:14,804-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:35:16,716-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:35:26,727-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:35:36,735-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:35:44,073-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:35:46,745-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 01:35:51,279-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 01:35:56,756-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:36:06,766-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:36:16,776-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 01:36:21,392-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 01:36:26,786-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:36:27,753-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:36:36,795-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:36:46,804-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:36:56,813-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:36:58,709-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:37:04,227-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:37:06,823-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:37:16,832-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:37:26,842-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:37:36,027-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:37:36,851-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 01:37:40,702-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 01:37:46,860-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:37:56,869-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:38:06,878-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 01:38:13,346-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 01:38:16,887-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 01:38:17,176-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 01:38:26,897-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:38:36,906-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:38:46,914-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 01:38:50,664-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:38:53,649-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:38:56,924-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:39:06,932-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:39:16,942-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:39:26,950-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 01:39:27,982-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:39:30,123-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:39:36,958-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:39:46,971-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:39:56,990-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:40:05,299-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 01:40:06,598-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 01:40:07,001-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:40:17,009-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:40:27,016-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:40:37,025-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 01:40:42,618-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:40:43,071-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:40:47,032-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:40:57,042-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:41:07,052-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:41:17,065-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:41:19,545-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:41:19,935-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:41:27,078-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:41:37,092-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:41:47,100-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 01:41:56,020-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 01:41:57,108-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:41:57,253-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:42:07,116-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:42:17,124-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:42:27,131-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:42:32,493-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:42:34,571-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:42:37,139-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:42:47,147-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:42:57,154-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:43:07,162-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:43:08,967-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:43:11,889-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:43:17,170-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:43:27,178-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:43:37,186-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:43:45,441-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:43:47,193-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:43:49,207-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:43:57,201-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:44:07,209-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:44:17,216-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 01:44:21,915-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:44:26,525-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:44:27,224-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:44:37,232-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:44:47,239-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:44:57,247-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:44:58,389-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:45:03,844-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:45:07,255-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:45:17,262-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:45:27,270-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 01:45:34,863-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 01:45:37,278-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:45:41,162-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:45:47,286-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:45:57,294-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:46:07,301-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:46:11,337-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:46:17,309-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:46:18,480-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:46:27,317-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:46:37,324-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:46:47,332-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 01:46:47,812-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:46:55,798-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:46:57,340-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:47:07,348-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:47:17,355-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:47:24,285-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:47:27,363-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:47:33,116-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:47:37,371-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:47:47,379-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:47:57,386-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:48:00,759-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:48:07,394-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:48:10,434-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:48:17,402-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:48:27,409-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:48:37,233-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:48:37,417-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:48:47,425-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:48:47,752-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:48:57,433-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:49:07,440-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:49:13,707-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:49:17,448-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:49:25,070-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:49:27,456-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:49:37,463-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:49:47,471-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 01:49:50,181-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 01:49:57,479-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:50:02,388-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:50:07,486-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:50:17,494-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:50:26,655-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:50:27,502-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:50:37,510-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:50:39,706-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:50:47,517-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:50:57,525-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 01:51:03,130-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 01:51:07,532-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:51:17,024-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:51:17,540-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:51:27,548-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:51:37,555-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:51:39,603-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:51:47,563-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:51:54,342-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:51:57,570-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:52:07,578-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 01:52:16,077-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 01:52:17,586-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:52:27,593-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:52:31,660-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:52:37,601-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:52:47,609-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:52:52,551-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:52:57,616-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:53:07,624-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:53:08,978-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:53:17,632-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:53:27,639-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:53:29,025-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:53:37,647-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:53:46,296-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:53:47,654-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:53:57,662-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:54:05,499-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:54:07,670-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:54:17,677-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:54:23,614-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:54:27,685-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:54:37,693-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:54:41,973-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:54:47,700-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:54:57,708-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:55:00,932-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:55:07,715-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:55:17,723-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:55:18,447-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:55:27,731-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:55:37,738-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:55:38,250-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:55:47,746-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:55:54,921-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:55:57,753-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:56:07,761-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:56:15,568-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:56:17,768-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:56:27,776-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:56:31,395-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:56:37,784-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:56:47,791-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:56:52,886-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:56:57,799-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:57:07,807-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 99ms
-2013-11-07 01:57:07,967-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 99ms
-No streams
-2013-11-07 01:57:17,815-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:57:27,822-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:57:30,204-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:57:37,830-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-07 01:57:44,343-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-07 01:57:47,838-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:57:57,845-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:58:07,522-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:58:07,853-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:58:17,860-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:58:20,818-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:58:27,868-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:58:37,876-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:58:44,840-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:58:47,884-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 01:58:57,293-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 01:58:57,891-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:59:07,899-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:59:17,907-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:59:22,158-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:59:27,914-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:59:33,766-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 01:59:37,922-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:59:47,930-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 01:59:57,938-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 01:59:59,476-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:00:07,945-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:00:10,240-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:00:17,953-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:00:27,961-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:00:36,795-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:00:37,968-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:00:46,714-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:00:47,976-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:00:57,984-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:01:07,992-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:01:14,112-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:01:18,000-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 02:01:23,188-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 02:01:28,007-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:01:38,015-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:01:48,023-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:01:51,430-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:01:58,031-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:01:59,662-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:02:08,040-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:02:18,048-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:02:28,056-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:02:28,748-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 02:02:36,136-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 02:02:38,064-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:02:48,072-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:02:58,080-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:03:06,066-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:03:08,089-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 02:03:12,611-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 02:03:18,097-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:03:28,105-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:03:38,113-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:03:43,384-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:03:48,122-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:03:49,084-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:03:58,130-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:04:08,138-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:04:18,146-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:04:20,702-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:04:25,558-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:04:28,155-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:04:38,163-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:04:48,171-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 02:04:58,020-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 02:04:58,179-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 02:05:02,032-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 02:05:08,187-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:05:18,195-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:05:28,203-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:05:35,339-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:05:38,212-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 02:05:38,507-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 02:05:48,220-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:05:58,228-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:06:08,236-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:06:12,657-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:06:14,980-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:06:18,244-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:06:28,252-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:06:38,261-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:06:48,269-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:06:49,975-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:06:51,454-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:06:58,277-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:07:08,285-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:07:18,293-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:07:27,293-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 02:07:27,929-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 02:07:28,301-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:07:38,309-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:07:48,317-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:07:58,325-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:08:04,403-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 02:08:04,611-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 02:08:08,333-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:08:18,341-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:08:28,349-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:08:38,357-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:08:40,877-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:08:41,929-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:08:48,365-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:08:58,373-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:09:08,382-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:09:17,352-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:09:18,390-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:09:19,247-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:09:28,398-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:09:38,407-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:09:48,415-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:09:53,825-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:09:56,565-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:09:58,423-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:10:08,431-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:10:18,439-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:10:28,447-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:10:30,300-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:10:33,883-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:10:38,455-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:10:48,463-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:10:58,471-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:11:06,774-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:11:08,479-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 02:11:11,201-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 02:11:18,487-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:11:28,495-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:11:38,503-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:11:43,248-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:11:48,512-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:11:48,519-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:11:58,520-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:12:08,528-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:12:18,536-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 02:12:19,723-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:12:25,837-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:12:28,544-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:12:38,551-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:12:48,559-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 02:12:56,196-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 02:12:58,567-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:13:03,155-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:13:08,575-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:13:18,583-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:13:28,591-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:13:32,671-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:13:38,599-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:13:40,473-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:13:48,607-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:13:58,614-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:14:08,622-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 02:14:09,146-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:14:17,791-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:14:18,631-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:14:28,639-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:14:38,647-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:14:45,619-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:14:48,655-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:14:55,109-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:14:58,663-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:15:08,671-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:15:18,679-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 02:15:22,094-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 02:15:28,687-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:15:32,427-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:15:38,695-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:15:48,704-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:15:58,567-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:15:58,712-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:16:08,719-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:16:09,745-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:16:18,727-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:16:28,735-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 02:16:35,042-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 02:16:38,742-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:16:47,063-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:16:48,750-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:16:58,758-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:17:08,766-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:17:11,515-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:17:18,774-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:17:24,381-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:17:28,782-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:17:38,790-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 02:17:47,990-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 02:17:48,798-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:17:58,806-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:18:01,699-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:18:08,814-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:18:18,822-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:18:24,463-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:18:28,829-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:18:38,837-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 02:18:39,017-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 02:18:48,845-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:18:58,853-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:19:00,937-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:19:08,861-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:19:16,336-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:19:18,870-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:19:28,878-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:19:37,411-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:19:38,886-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:19:48,894-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 02:19:53,654-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 02:19:58,902-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:20:08,910-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:20:13,885-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:20:18,918-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:20:28,926-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:20:30,972-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:20:38,934-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:20:48,942-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:20:50,359-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:20:58,950-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:21:08,290-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:21:08,958-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:21:18,966-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:21:26,833-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:21:28,974-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:21:38,982-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:21:45,608-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:21:48,990-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:21:58,998-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:22:03,307-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:22:09,006-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:22:19,015-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:22:22,926-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:22:29,023-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:22:39,031-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:22:39,781-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:22:49,039-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:22:59,047-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 02:23:00,244-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 02:23:09,055-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 02:23:16,254-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 02:23:19,063-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:23:29,071-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:23:37,562-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:23:39,079-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:23:49,087-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:23:52,729-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:23:59,095-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:24:09,103-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 02:24:14,880-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 02:24:19,111-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:24:29,120-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:24:29,203-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:24:39,128-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:24:49,136-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:24:52,198-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:24:59,144-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 02:25:05,678-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 02:25:09,152-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:25:19,160-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:25:29,168-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:25:29,516-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:25:39,176-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 02:25:42,151-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 02:25:49,184-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:25:59,193-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 02:26:06,833-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 02:26:09,200-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 02:26:18,626-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 02:26:19,208-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:26:29,216-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:26:39,224-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:26:44,152-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:26:49,232-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 02:26:55,100-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 02:26:59,240-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:27:09,248-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:27:19,256-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 02:27:21,470-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 02:27:29,264-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:27:31,573-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:27:39,272-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:27:49,280-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:27:58,788-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:27:59,288-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 02:28:08,048-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 02:28:09,297-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:28:19,305-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:28:29,313-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 02:28:36,106-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 02:28:39,321-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:28:44,522-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:28:49,329-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:28:59,337-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:29:09,346-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:29:13,424-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:29:19,354-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 02:29:20,997-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 02:29:29,362-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:29:39,370-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:29:49,378-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:29:50,742-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:29:57,470-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:29:59,386-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:30:09,394-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:30:19,402-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 02:30:28,060-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 02:30:29,410-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 02:30:33,945-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 02:30:39,418-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:30:49,426-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:30:59,434-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:31:05,378-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:31:09,442-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:31:10,418-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:31:19,450-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:31:29,458-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:31:39,467-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 02:31:42,696-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:31:46,892-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:31:49,475-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:31:59,483-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:32:09,491-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:32:19,499-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:32:20,014-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:32:23,366-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:32:29,507-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:32:39,515-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:32:49,523-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:32:57,332-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:32:59,531-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 02:32:59,841-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 02:33:09,539-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:33:19,547-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:33:29,555-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 02:33:34,651-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 02:33:36,314-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 02:33:39,563-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:33:49,571-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:33:59,579-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:34:09,587-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:34:11,969-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 02:34:12,789-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 02:34:19,595-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:34:29,603-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:34:39,611-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 02:34:49,263-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 02:34:49,287-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 02:34:49,620-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:34:59,628-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:35:09,636-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:35:19,644-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 02:35:25,737-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 02:35:26,605-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 02:35:29,652-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:35:39,660-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:35:49,668-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:35:59,677-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 02:36:02,210-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:36:03,923-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:36:09,685-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:36:19,693-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:36:29,701-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 02:36:38,685-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 02:36:39,709-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 02:36:41,241-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 02:36:49,717-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:36:59,724-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:37:09,732-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:37:15,158-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:37:18,559-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:37:19,740-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:37:29,748-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:37:39,756-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:37:49,764-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:37:51,632-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 02:37:55,877-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 02:37:59,772-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:38:09,780-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:38:19,788-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:38:28,106-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:38:29,796-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 02:38:33,195-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 02:38:39,804-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:38:49,812-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:38:59,820-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 02:39:04,581-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 02:39:09,828-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 02:39:10,514-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 02:39:19,837-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:39:29,845-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:39:39,853-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 02:39:41,054-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 02:39:47,832-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 02:39:49,861-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:39:59,869-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:40:09,878-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:40:17,528-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:40:19,886-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:40:25,150-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:40:29,894-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:40:39,902-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:40:49,911-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:40:54,002-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:40:59,919-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 02:41:02,468-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 02:41:09,927-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:41:19,935-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:41:29,943-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:41:30,476-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:41:39,786-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:41:39,952-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:41:49,960-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:41:59,968-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 02:42:06,949-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 02:42:09,976-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 02:42:17,104-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 02:42:19,984-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:42:29,992-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:42:40,001-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:42:43,424-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:42:50,009-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:42:54,422-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:43:00,017-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:43:10,025-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:43:19,899-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:43:20,033-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:43:30,041-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 02:43:31,740-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 02:43:40,049-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:43:50,057-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:43:56,373-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:44:00,066-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 02:44:09,058-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 02:44:10,074-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:44:20,082-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:44:30,090-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:44:32,847-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:44:40,098-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:44:46,377-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:44:50,107-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:45:00,115-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:45:09,321-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:45:10,123-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:45:20,131-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 02:45:23,695-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 02:45:30,139-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:45:40,147-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 02:45:45,796-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 02:45:50,155-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:46:00,163-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:46:01,013-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:46:10,171-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:46:20,179-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:46:22,269-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:46:30,187-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 02:46:38,331-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 02:46:40,195-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:46:50,203-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 02:46:58,744-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 02:47:00,211-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:47:10,219-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 02:47:15,649-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 02:47:20,227-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:47:30,235-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:47:35,218-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:47:40,243-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:47:50,251-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 02:47:52,968-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 02:48:00,259-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:48:10,267-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 02:48:11,692-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 02:48:20,275-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:48:30,283-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 02:48:30,286-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 02:48:40,291-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 02:48:48,165-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 02:48:50,299-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:49:00,307-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:49:07,604-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:49:10,315-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:49:20,323-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 02:49:24,640-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 02:49:30,331-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:49:40,339-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 02:49:44,922-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 02:49:50,347-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:50:00,355-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 02:50:01,113-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 02:50:10,363-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:50:20,371-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:50:22,240-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:50:30,379-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:50:37,587-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:50:40,387-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:50:50,395-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 02:50:59,558-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 02:51:00,403-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:51:10,411-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 02:51:14,061-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 02:51:20,419-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:51:30,426-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 02:51:36,876-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 02:51:40,434-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:51:50,442-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:51:50,535-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:52:00,450-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:52:10,458-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:52:14,194-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:52:20,466-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 02:52:27,009-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 02:52:30,474-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:52:40,482-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:52:50,490-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 02:52:51,512-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 02:53:00,498-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 02:53:03,483-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 02:53:10,506-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:53:20,514-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:53:28,830-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:53:30,522-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 02:53:39,957-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 02:53:40,530-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:53:50,537-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:54:00,545-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 02:54:06,148-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 02:54:10,553-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:54:16,431-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:54:20,561-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:54:30,569-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:54:40,577-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:54:43,466-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:54:50,585-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 02:54:52,904-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 02:55:00,593-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:55:10,601-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:55:20,609-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 02:55:20,784-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 02:55:29,380-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 02:55:30,617-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:55:40,625-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:55:50,633-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 02:55:58,102-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 02:56:00,641-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 02:56:05,853-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 02:56:10,649-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:56:20,657-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:56:30,665-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:56:35,420-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:56:40,673-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:56:42,327-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:56:50,681-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:57:00,689-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:57:10,697-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 02:57:12,738-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 02:57:18,801-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 02:57:20,705-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:57:30,713-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:57:40,721-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 02:57:50,056-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 02:57:50,729-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 02:57:55,276-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 02:58:00,737-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:58:10,745-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:58:20,752-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 02:58:27,376-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 02:58:30,760-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 02:58:31,749-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 02:58:40,768-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:58:50,776-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:59:00,784-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 02:59:04,693-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:59:08,223-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 02:59:10,792-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:59:20,800-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:59:30,808-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 02:59:40,816-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 02:59:42,011-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 02:59:44,697-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 02:59:50,824-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:00:00,833-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:00:10,841-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:00:19,329-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 03:00:20,849-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 03:00:21,172-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 03:00:30,857-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:00:40,865-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:00:50,873-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:00:56,647-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:00:57,645-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 03:01:00,881-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:01:10,889-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:01:20,897-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:01:30,905-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:01:33,965-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:01:34,119-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:01:40,913-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:01:50,921-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:02:00,929-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:02:10,593-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 03:02:10,937-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:02:11,283-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 03:02:20,945-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:02:30,953-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:02:40,961-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:02:47,067-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 03:02:48,600-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 03:02:50,969-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:03:00,977-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:03:10,985-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:03:20,993-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 03:03:23,540-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:03:25,919-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:03:31,001-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:03:41,009-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:03:51,017-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 03:04:00,016-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 03:04:01,025-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:04:03,237-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 03:04:11,033-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:04:21,041-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:04:31,049-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:04:36,489-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:04:40,555-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:04:41,057-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:04:51,065-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:05:01,073-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:05:11,081-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:05:12,963-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:05:17,873-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 03:05:21,089-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:05:31,097-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:05:41,105-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:05:49,437-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 03:05:51,113-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 03:05:55,191-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 03:06:01,121-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:06:11,129-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:06:21,137-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:06:25,911-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:06:31,145-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:06:32,510-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 03:06:41,153-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:06:51,161-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:07:01,169-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:07:02,385-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 03:07:09,827-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 03:07:11,177-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:07:21,185-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:07:31,193-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:07:38,859-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 03:07:41,201-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:07:47,146-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:07:51,209-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:08:01,217-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:08:11,225-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:08:15,333-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 03:08:21,233-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:08:24,464-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 03:08:31,241-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:08:41,249-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:08:51,257-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:08:51,807-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:09:01,265-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:09:01,782-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:09:11,273-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:09:21,281-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 03:09:28,280-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 03:09:31,289-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:09:39,100-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 03:09:41,296-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:09:51,304-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:10:01,312-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:10:04,755-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:10:11,320-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:10:16,418-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 03:10:21,328-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:10:31,336-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 03:10:41,230-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 03:10:41,344-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:10:51,352-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:10:53,736-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:11:01,360-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:11:11,368-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:11:17,703-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:11:21,376-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:11:31,054-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 03:11:31,384-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:11:41,392-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:11:51,400-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 03:11:54,176-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 03:12:01,408-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:12:08,372-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:12:11,416-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:12:21,424-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:12:30,651-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:12:31,431-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:12:41,439-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:12:45,690-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 03:12:51,447-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:13:01,455-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:13:07,125-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:13:11,463-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:13:21,471-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:13:23,008-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:13:31,479-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:13:41,487-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:13:43,599-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 03:13:51,495-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:14:00,326-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 03:14:01,503-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:14:11,510-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:14:20,073-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:14:21,518-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:14:31,526-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 03:14:37,643-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 03:14:41,534-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:14:51,542-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:14:56,547-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:15:01,550-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:15:11,558-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 03:15:14,963-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 03:15:21,565-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:15:31,573-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:15:33,021-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:15:41,581-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:15:51,589-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:15:52,280-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:16:01,596-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:16:09,495-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:16:11,604-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:16:21,612-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:16:29,598-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 03:16:31,620-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:16:41,628-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 5ms
-2013-11-07 03:16:45,973-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 5ms
-No streams
-2013-11-07 03:16:51,636-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:17:01,643-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:17:06,916-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:17:11,651-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:17:21,659-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:17:22,443-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:17:31,667-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:17:41,675-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:17:44,234-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 03:17:51,682-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:17:58,917-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 03:18:01,690-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:18:11,698-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:18:21,552-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 03:18:21,706-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:18:31,714-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:18:35,391-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 03:18:41,721-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:18:51,729-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 03:18:58,869-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 03:19:01,737-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:19:11,745-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:19:11,865-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:19:21,752-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:19:31,760-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:19:36,188-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:19:41,768-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 03:19:48,340-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 03:19:51,776-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:20:01,783-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:20:11,791-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:20:13,506-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 03:20:21,799-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:20:24,813-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 03:20:31,807-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:20:41,814-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:20:50,824-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 03:20:51,822-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 03:21:01,288-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 03:21:01,830-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:21:11,838-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:21:21,846-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 03:21:28,142-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 03:21:31,854-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:21:37,761-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 03:21:41,862-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:21:51,870-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:22:01,878-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:22:05,460-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:22:11,886-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:22:14,235-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:22:21,893-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:22:31,901-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:22:41,909-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:22:42,778-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:22:50,709-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:22:51,917-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:23:01,925-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:23:11,933-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:23:20,096-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 03:23:21,941-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:23:27,183-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 03:23:31,949-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:23:41,956-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:23:51,964-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:23:57,414-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:24:01,972-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:24:03,657-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:24:11,980-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:24:21,987-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:24:31,995-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:24:34,732-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 03:24:40,130-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 03:24:42,003-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:24:52,011-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:25:02,019-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:25:12,026-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:25:12,051-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:25:16,605-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:25:22,034-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:25:32,042-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:25:42,050-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:25:49,369-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:25:52,058-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:25:53,079-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:26:02,066-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:26:12,073-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:26:22,081-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:26:26,687-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:26:29,553-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:26:32,089-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:26:42,097-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:26:52,104-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:27:02,112-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:27:04,005-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:27:06,028-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:27:12,121-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:27:22,129-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:27:32,137-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:27:41,323-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:27:42,145-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:27:42,503-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:27:52,154-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:28:02,162-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:28:12,170-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:28:18,641-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:28:18,976-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 03:28:22,178-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:28:32,187-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:28:42,195-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:28:52,203-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:28:55,450-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:28:55,959-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:29:02,213-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:29:12,221-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:29:22,229-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:29:31,924-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:29:32,239-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:29:33,278-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 03:29:42,248-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:29:52,257-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:30:02,277-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:30:08,398-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:30:10,596-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 03:30:12,296-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:30:22,302-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:30:32,321-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:30:42,330-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:30:44,872-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 03:30:47,914-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 03:30:52,340-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:31:02,350-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:31:12,360-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:31:21,346-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:31:22,368-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 03:31:25,232-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 03:31:32,377-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:31:42,387-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:31:52,398-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 03:31:57,821-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 03:32:02,408-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:32:02,550-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 03:32:12,418-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:32:22,426-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:32:32,435-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:32:34,295-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 03:32:39,868-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 03:32:42,443-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:32:52,452-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:33:02,460-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:33:10,769-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:33:12,468-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:33:17,185-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 03:33:22,477-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:33:32,488-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:33:42,506-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:33:47,243-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:33:52,516-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:33:54,503-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:34:02,525-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:34:12,535-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:34:22,544-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 03:34:23,718-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:34:31,821-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 03:34:32,552-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:34:42,561-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:34:52,571-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:35:00,191-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:35:02,580-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:35:09,140-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 03:35:12,589-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:35:22,598-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:35:32,605-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 03:35:36,666-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 03:35:42,613-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:35:46,458-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 03:35:52,621-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:36:02,629-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:36:12,638-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:36:13,139-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:36:22,646-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 03:36:23,777-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 03:36:32,656-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:36:42,666-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 03:36:49,614-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 03:36:52,676-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 03:37:01,095-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 03:37:02,687-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:37:12,698-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:37:22,707-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 03:37:26,088-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 03:37:32,715-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 03:37:38,413-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 03:37:42,724-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:37:52,734-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:38:02,561-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:38:02,745-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:38:12,755-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:38:15,731-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 03:38:22,764-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:38:32,774-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 03:38:39,036-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 03:38:42,783-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:38:52,792-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:38:53,048-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:39:02,802-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:39:12,811-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:39:15,509-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:39:22,820-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 03:39:30,367-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 03:39:32,829-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:39:42,839-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 03:39:51,984-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 03:39:52,848-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:40:02,856-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:40:07,684-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:40:12,868-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:40:22,882-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 03:40:28,458-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 03:40:32,893-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:40:42,902-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:40:45,002-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:40:52,912-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:41:02,921-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:41:04,931-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:41:12,928-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:41:22,320-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:41:22,936-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:41:32,944-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:41:41,405-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:41:42,952-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:41:52,960-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:41:59,638-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 03:42:02,968-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:42:12,976-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:42:17,879-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:42:22,983-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:42:32,991-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:42:36,956-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:42:42,999-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:42:53,007-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:42:54,353-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:43:03,015-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:43:13,026-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:43:14,274-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 03:43:23,034-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:43:30,827-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:43:33,042-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:43:43,050-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:43:51,592-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:43:53,058-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:44:03,066-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:44:07,301-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:44:13,074-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:44:23,082-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:44:28,910-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:44:33,090-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:44:43,097-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:44:43,775-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:44:53,105-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:45:03,113-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:45:06,228-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 03:45:13,121-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:45:20,249-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 03:45:23,129-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:45:33,137-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:45:43,145-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:45:43,546-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:45:53,153-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:45:56,723-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:46:03,161-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:46:13,168-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:46:20,864-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:46:23,176-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:46:33,184-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:46:33,197-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:46:43,192-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:46:53,200-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:46:58,182-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 03:47:03,208-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:47:09,671-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:47:13,216-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:47:23,224-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:47:33,232-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:47:35,500-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:47:43,240-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:47:46,145-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:47:53,248-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:48:03,256-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:48:12,818-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 03:48:13,264-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:48:22,619-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:48:23,272-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:48:33,280-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:48:43,288-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:48:50,136-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:48:53,296-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:48:59,093-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:49:03,303-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:49:13,311-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:49:23,319-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:49:27,454-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:49:33,327-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:49:35,567-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:49:43,335-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:49:53,343-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:50:03,351-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:50:04,773-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 03:50:12,041-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 03:50:13,359-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:50:23,367-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:50:33,376-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:50:42,091-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:50:43,384-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:50:48,516-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:50:53,392-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:51:03,400-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:51:13,408-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:51:19,409-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 03:51:23,416-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:51:24,990-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:51:33,424-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:51:43,432-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:51:53,440-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:51:56,727-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:52:01,464-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:52:03,448-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:52:13,456-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:52:23,465-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:52:33,473-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:52:34,045-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:52:37,938-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:52:43,481-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:52:53,489-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:53:03,497-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:53:11,363-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 03:53:13,505-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:53:14,412-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:53:23,513-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:53:33,521-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:53:43,529-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:53:48,681-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 03:53:50,885-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 03:53:53,537-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:54:03,546-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:54:13,554-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:54:23,562-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:54:25,999-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:54:27,360-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:54:33,570-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:54:43,578-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:54:53,586-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:55:03,318-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:55:03,594-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:55:03,835-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:55:13,602-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:55:23,610-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:55:33,618-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:55:40,309-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:55:40,635-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:55:43,626-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:55:53,634-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:56:03,642-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:56:13,650-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:56:16,783-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:56:17,953-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 03:56:23,657-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:56:33,665-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:56:43,673-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:56:53,257-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:56:53,681-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:56:55,271-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:57:03,689-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:57:13,697-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:57:23,705-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:57:29,731-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:57:32,589-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:57:33,713-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:57:43,721-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:57:53,729-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:58:03,736-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:58:06,205-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:58:09,908-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 03:58:13,744-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:58:23,752-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:58:33,760-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:58:42,679-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:58:43,768-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:58:47,226-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:58:53,776-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:59:03,783-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:59:13,791-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:59:19,153-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 03:59:23,799-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 03:59:24,544-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 03:59:33,807-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:59:43,815-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 03:59:53,823-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 03:59:55,627-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:00:01,862-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:00:03,831-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:00:13,839-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:00:23,847-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:00:32,101-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:00:33,855-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 04:00:39,180-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 04:00:43,863-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:00:53,871-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:01:03,879-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:01:08,575-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:01:13,888-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:01:16,498-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:01:23,896-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:01:33,904-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:01:43,912-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:01:45,049-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 04:01:53,816-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 04:01:53,921-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:02:03,929-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:02:13,937-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:02:21,523-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:02:23,945-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:02:31,134-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:02:33,953-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:02:43,961-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:02:53,970-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:02:57,997-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:03:03,978-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 04:03:08,452-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 04:03:13,986-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:03:23,994-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:03:34,003-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:03:34,471-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:03:44,011-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:03:45,770-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:03:54,019-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:04:04,027-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 04:04:10,944-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 04:04:14,035-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:04:23,088-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:04:24,044-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:04:34,052-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:04:44,060-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:04:47,419-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:04:54,068-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 04:05:00,406-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 04:05:04,076-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:05:14,084-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:05:23,894-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:05:24,093-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:05:34,101-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:05:37,724-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:05:44,109-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:05:54,118-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:06:00,368-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:06:04,126-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:06:14,134-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:06:15,042-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:06:24,142-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:06:34,151-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 04:06:36,841-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 04:06:44,159-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 04:06:52,360-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 04:06:54,167-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:07:04,175-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:07:13,316-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:07:14,183-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:07:24,192-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:07:29,678-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:07:34,200-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:07:44,208-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 04:07:49,789-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 04:07:54,216-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:08:04,224-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 04:08:06,996-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 04:08:14,231-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:08:24,239-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:08:26,264-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:08:34,247-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:08:44,255-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:08:44,314-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:08:54,263-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 04:09:02,740-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 04:09:04,271-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:09:14,279-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:09:21,632-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:09:24,287-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:09:34,295-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:09:39,213-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:09:44,302-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:09:54,310-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 04:09:58,951-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 04:10:04,318-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:10:14,326-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 04:10:15,688-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 04:10:24,334-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:10:34,342-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:10:36,269-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:10:44,349-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:10:52,162-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:10:54,357-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:11:04,365-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 04:11:13,587-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 04:11:14,373-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:11:24,381-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 04:11:28,636-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 04:11:34,389-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:11:44,396-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:11:50,905-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:11:54,404-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:12:04,412-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:12:05,110-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:12:14,420-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:12:24,428-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 04:12:28,223-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 04:12:34,436-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:12:41,584-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:12:44,444-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:12:54,452-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:13:04,460-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:13:05,541-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:13:14,467-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:13:18,058-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:13:24,475-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:13:34,483-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 04:13:42,859-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 04:13:44,491-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:13:54,499-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:13:54,532-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:14:04,507-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:14:14,515-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:14:20,177-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:14:24,523-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:14:31,006-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:14:34,531-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:14:44,539-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:14:54,546-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:14:57,495-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:15:04,554-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:15:07,480-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:15:14,562-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:15:24,570-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:15:34,578-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 04:15:34,814-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:15:43,954-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:15:44,586-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:15:54,594-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:16:04,602-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:16:12,132-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:16:14,610-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:16:20,428-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:16:24,617-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:16:34,625-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:16:44,633-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 04:16:49,450-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 04:16:54,641-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:16:56,902-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:17:04,649-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:17:14,657-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:17:24,664-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:17:26,768-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:17:33,376-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:17:34,672-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:17:44,680-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:17:54,688-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:18:04,086-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:18:04,696-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:18:09,850-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:18:14,704-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:18:24,712-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:18:34,719-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 04:18:41,404-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 04:18:44,727-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:18:46,324-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:18:54,735-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:19:04,743-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:19:14,751-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 04:19:18,723-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 04:19:22,798-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 04:19:24,759-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:19:34,766-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:19:44,774-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:19:54,782-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:19:56,041-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:19:59,273-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:20:04,790-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:20:14,798-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:20:24,806-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:20:33,359-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:20:34,814-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 04:20:35,747-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 04:20:44,822-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:20:54,830-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:21:04,838-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 04:21:10,677-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:21:12,222-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:21:14,845-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:21:24,853-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:21:34,861-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:21:44,870-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:21:47,995-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 04:21:48,696-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 04:21:54,878-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:22:04,886-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:22:14,894-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:22:24,902-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:22:25,171-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:22:25,313-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:22:34,910-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:22:44,918-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:22:54,926-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:23:01,645-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 04:23:02,631-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 04:23:04,934-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:23:14,942-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:23:24,950-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:23:34,958-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:23:38,119-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:23:39,949-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:23:44,966-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:23:54,974-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:24:04,982-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:24:14,593-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:24:14,990-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 04:24:17,267-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 04:24:24,999-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:24:35,007-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:24:45,015-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 92ms
-2013-11-07 04:24:51,158-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 92ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:24:54,585-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:24:55,023-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:25:05,031-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:25:15,039-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:25:25,047-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 04:25:27,542-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:25:31,903-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:25:35,055-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:25:45,063-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:25:55,071-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:26:04,015-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:26:05,079-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 04:26:09,221-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 04:26:15,088-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:26:25,095-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:26:35,103-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:26:40,489-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:26:45,111-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:26:46,539-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:26:55,119-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:27:05,127-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:27:15,135-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:27:16,963-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 04:27:23,857-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 04:27:25,143-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:27:35,150-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:27:45,158-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:27:53,437-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:27:55,166-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 04:28:01,176-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 04:28:05,174-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:28:15,182-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:28:25,190-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:28:29,911-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:28:35,198-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:28:38,493-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:28:45,205-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:28:55,213-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:29:05,221-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:29:06,385-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:29:15,229-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:29:15,811-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:29:25,237-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:29:35,245-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 04:29:42,858-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 04:29:45,252-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:29:53,129-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:29:55,260-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:30:05,268-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:30:15,276-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:30:19,333-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:30:25,284-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:30:30,447-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:30:35,292-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:30:45,299-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:30:55,307-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:30:55,807-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:31:05,315-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:31:07,765-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:31:15,323-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:31:25,331-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:31:32,281-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:31:35,339-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 04:31:45,083-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 04:31:45,347-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:31:55,355-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:32:05,363-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:32:08,755-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:32:15,371-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:32:22,401-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:32:25,379-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:32:35,388-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:32:45,229-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:32:45,395-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:32:55,403-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:32:59,719-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:33:05,411-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:33:15,420-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:33:21,703-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:33:25,428-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:33:35,436-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 04:33:37,037-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 04:33:45,444-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:33:55,452-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:33:58,177-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:34:05,460-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:34:14,355-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:34:15,468-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:34:25,475-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:34:34,651-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:34:35,483-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:34:45,491-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 04:34:51,673-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 04:34:55,498-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:35:05,506-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:35:11,125-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:35:15,514-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:35:25,522-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 04:35:28,992-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 04:35:35,530-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:35:45,538-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:35:47,599-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:35:55,546-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:36:05,553-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:36:06,309-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:36:15,561-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:36:24,073-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:36:25,569-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:36:35,576-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 04:36:43,627-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 04:36:45,584-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:36:55,592-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:37:00,547-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:37:05,600-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:37:15,608-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:37:20,945-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:37:25,615-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:37:35,623-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:37:37,021-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:37:45,631-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:37:55,639-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 04:37:58,263-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 04:38:05,647-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:38:13,496-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:38:15,654-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:38:25,662-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:38:35,581-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:38:35,670-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:38:45,678-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:38:49,970-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:38:55,686-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:39:05,694-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:39:12,899-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:39:15,702-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:39:25,710-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:39:26,444-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:39:35,718-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:39:45,727-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 04:39:50,217-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 04:39:55,735-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:40:02,918-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:40:05,743-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:40:15,752-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:40:25,760-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:40:27,536-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:40:35,768-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 20ms
-2013-11-07 04:40:39,411-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 20ms
-No streams
-2013-11-07 04:40:45,776-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:40:55,784-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 04:41:04,854-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 04:41:05,793-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:41:15,801-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:41:15,866-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:41:25,809-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:41:35,817-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:41:42,172-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:41:45,825-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-07 04:41:52,339-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-07 04:41:55,833-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:42:05,841-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:42:15,849-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 04:42:19,490-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 04:42:25,857-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:42:28,814-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:42:35,866-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:42:45,874-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:42:55,882-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:42:56,808-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:43:05,288-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:43:05,890-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:43:15,898-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:43:25,906-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 04:43:34,126-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 04:43:35,915-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:43:41,762-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:43:45,923-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:43:55,931-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:44:05,939-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:44:11,444-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:44:15,947-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:44:18,236-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:44:25,955-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:44:35,964-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:44:45,972-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 04:44:48,762-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 04:44:54,709-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 04:44:55,980-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:45:05,988-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:45:15,996-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:45:26,004-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 04:45:26,081-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:45:31,184-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:45:36,012-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:45:46,020-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:45:56,029-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:46:03,398-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:46:06,037-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:46:07,658-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:46:16,045-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:46:26,053-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:46:36,061-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 04:46:40,716-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:46:44,132-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:46:46,068-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:46:56,076-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:47:06,084-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:47:16,092-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:47:18,034-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 04:47:20,605-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 04:47:26,100-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:47:36,108-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:47:46,116-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 04:47:55,352-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 04:47:56,124-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:47:57,080-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:48:06,132-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:48:16,139-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:48:26,147-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:48:32,670-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 04:48:33,554-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 04:48:36,155-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:48:46,163-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:48:56,171-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:49:06,179-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 04:49:09,988-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:49:10,028-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:49:16,187-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:49:26,195-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:49:36,203-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:49:46,211-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:49:46,502-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:49:47,306-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:49:56,219-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:50:06,227-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:50:16,235-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:50:22,976-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:50:24,624-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:50:26,243-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:50:36,251-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:50:46,259-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:50:56,267-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:50:59,450-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 04:51:01,941-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 04:51:06,275-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:51:16,283-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:51:26,291-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:51:35,924-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:51:36,298-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:51:39,260-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:51:46,306-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:51:56,314-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:52:06,322-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:52:12,398-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:52:16,330-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 04:52:16,578-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 04:52:26,338-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:52:36,345-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:52:46,353-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 04:52:48,872-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:52:53,896-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:52:56,360-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:53:06,368-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:53:16,376-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:53:25,347-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:53:26,384-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 04:53:31,214-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 04:53:36,392-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:53:46,399-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:53:56,407-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 04:54:01,822-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 04:54:06,415-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:54:08,532-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:54:16,423-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:54:26,431-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:54:36,439-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:54:38,295-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 04:54:45,850-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 04:54:46,447-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:54:56,455-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:55:06,463-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 04:55:14,770-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 04:55:16,471-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:55:23,168-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:55:26,478-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:55:36,486-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:55:46,494-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:55:51,243-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:55:56,502-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:56:00,486-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:56:06,511-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:56:16,519-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:56:26,527-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:56:27,717-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:56:36,535-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:56:37,804-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:56:46,543-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:56:56,551-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 04:57:04,190-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 04:57:06,559-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 04:57:15,122-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 04:57:16,567-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:57:26,575-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:57:36,583-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:57:40,665-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:57:46,591-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:57:52,440-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:57:56,599-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:58:06,607-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:58:16,615-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:58:17,139-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:58:26,623-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 04:58:29,758-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 04:58:36,631-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:58:46,639-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:58:53,613-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:58:56,647-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:59:06,654-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 04:59:07,076-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 04:59:16,662-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:59:26,670-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:59:30,087-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:59:36,678-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 04:59:44,395-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 04:59:46,686-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 04:59:56,693-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:00:06,561-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:00:06,701-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:00:16,709-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:00:21,713-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:00:26,717-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:00:36,725-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 05:00:43,036-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 05:00:46,733-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:00:56,740-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:00:59,031-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:01:06,748-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:01:16,756-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:01:19,509-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:01:26,763-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:01:36,349-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:01:36,771-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:01:46,779-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:01:55,983-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:01:56,786-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:02:06,794-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 05:02:13,667-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 05:02:16,802-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:02:26,810-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:02:32,457-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:02:36,817-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:02:46,825-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:02:50,985-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:02:56,833-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:03:06,840-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:03:08,931-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:03:16,848-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:03:26,856-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:03:28,303-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:03:36,864-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:03:45,405-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:03:46,871-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:03:56,879-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:04:05,621-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:04:06,887-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:04:16,895-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:04:21,879-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:04:26,902-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:04:36,910-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:04:42,939-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:04:46,918-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:04:56,925-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:04:58,353-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:05:06,933-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:05:16,941-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:05:20,258-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:05:26,949-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 05:05:34,828-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 05:05:36,956-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:05:46,964-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:05:56,972-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:05:57,576-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:06:06,979-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:06:11,301-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:06:16,987-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:06:26,995-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:06:34,894-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:06:37,003-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:06:47,010-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:06:47,775-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:06:57,018-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:07:07,026-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 05:07:12,212-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 05:07:17,033-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:07:24,249-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:07:27,041-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:07:37,049-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:07:47,057-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:07:49,531-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:07:57,065-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 05:08:00,723-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 05:08:07,072-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:08:17,080-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 05:08:26,848-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 05:08:27,088-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:08:37,096-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:08:37,198-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:08:47,104-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:08:57,112-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:09:04,167-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:09:07,120-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:09:13,672-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:09:17,128-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:09:27,136-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:09:37,143-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:09:41,485-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:09:47,151-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:09:50,146-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:09:57,159-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:10:07,167-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:10:17,174-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 05:10:18,803-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:10:26,620-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:10:27,182-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:10:37,190-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:10:47,198-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:10:56,122-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:10:57,206-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:11:03,094-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:11:07,213-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:11:17,221-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:11:27,229-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:11:33,440-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:11:37,236-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 05:11:39,567-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 05:11:47,244-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:11:57,251-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:12:07,259-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:12:10,758-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:12:16,042-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:12:17,267-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:12:27,275-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:12:37,283-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:12:47,291-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:12:48,076-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:12:52,516-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:12:57,299-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:13:07,308-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:13:17,316-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 05:13:25,394-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 05:13:27,324-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 05:13:28,991-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 05:13:37,332-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:13:47,339-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:13:57,347-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:14:02,712-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 05:14:05,463-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 05:14:07,355-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:14:17,362-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:14:27,370-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:14:37,378-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:14:40,030-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:14:41,938-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:14:47,386-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:14:57,394-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:15:07,402-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 05:15:17,349-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 05:15:17,410-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:15:18,412-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:15:27,418-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:15:37,426-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:15:47,434-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:15:54,666-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:15:54,886-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:15:57,441-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:16:07,449-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:16:17,457-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:16:27,466-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:16:31,360-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:16:31,984-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:16:37,474-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:16:47,481-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:16:57,489-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:17:07,497-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:17:07,834-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 05:17:09,301-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 05:17:17,505-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:17:27,513-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:17:37,520-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:17:44,308-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:17:46,620-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:17:47,528-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:17:57,536-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:18:07,544-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:18:17,552-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:18:20,782-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:18:23,938-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:18:27,559-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:18:37,567-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:18:47,575-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:18:57,256-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:18:57,583-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:19:01,256-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:19:07,591-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:19:17,599-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:19:27,606-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:19:33,730-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:19:37,614-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:19:38,574-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:19:47,622-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:19:57,630-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:20:07,638-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:20:10,204-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 05:20:15,892-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 05:20:17,645-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:20:27,653-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:20:37,661-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:20:46,678-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:20:47,669-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:20:53,211-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:20:57,677-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:21:07,684-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:21:17,692-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:21:23,152-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:21:27,700-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:21:30,529-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:21:37,708-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:21:47,716-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:21:57,724-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:21:59,626-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:22:07,731-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:22:07,847-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:22:17,739-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:22:27,747-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:22:36,100-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:22:37,755-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:22:45,165-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:22:47,763-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:22:57,770-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:23:07,778-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:23:12,574-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:23:17,786-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:23:22,483-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:23:27,794-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:23:37,802-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:23:47,810-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:23:49,048-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:23:57,818-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:23:59,801-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:24:07,826-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:24:17,833-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:24:25,522-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:24:27,841-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:24:37,119-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:24:37,849-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:24:47,857-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:24:57,865-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:25:01,996-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:25:07,873-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:25:14,437-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:25:17,881-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:25:27,889-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:25:37,896-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:25:38,470-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:25:47,904-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:25:51,755-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:25:57,912-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:26:07,920-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:26:14,944-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:26:17,928-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:26:27,936-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:26:29,073-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:26:37,944-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:26:47,952-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:26:51,418-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:26:57,960-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:27:06,391-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:27:07,968-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:27:17,976-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:27:27,892-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:27:27,984-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:27:37,992-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:27:43,709-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:27:48,000-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:27:58,008-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:28:04,366-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:28:08,016-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:28:18,024-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:28:21,027-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:28:28,033-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:28:38,041-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 05:28:40,840-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 05:28:48,049-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:28:58,057-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:28:58,345-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:29:08,065-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:29:17,315-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:29:18,074-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:29:28,082-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:29:35,663-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:29:38,090-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:29:48,098-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:29:53,789-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:29:58,106-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:30:08,114-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:30:12,981-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:30:18,122-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:30:28,130-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:30:30,263-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:30:38,139-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:30:48,147-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:30:50,299-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:30:58,155-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:31:06,737-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:31:08,163-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:31:18,171-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:31:27,617-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:31:28,179-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:31:38,187-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:31:43,211-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:31:48,195-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:31:58,203-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:32:04,935-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:32:08,211-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:32:18,219-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 05:32:19,684-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 05:32:28,227-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:32:38,235-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:32:42,253-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:32:48,243-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:32:56,159-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:32:58,251-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:33:08,259-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:33:18,267-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:33:19,571-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:33:28,275-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:33:32,633-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:33:38,283-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:33:48,291-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:33:56,889-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:33:58,300-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:34:08,308-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:34:09,107-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:34:18,316-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:34:28,324-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 05:34:34,207-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 05:34:38,332-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:34:45,581-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:34:48,340-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:34:58,348-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:35:08,356-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:35:11,526-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:35:18,364-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:35:22,055-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:35:28,372-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:35:38,380-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:35:48,388-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:35:48,844-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:35:58,396-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 05:35:58,530-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 05:36:08,405-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:36:18,413-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:36:26,162-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:36:28,421-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:36:35,003-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:36:38,428-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:36:48,436-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:36:58,444-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:37:03,480-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:37:08,452-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:37:11,477-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:37:18,460-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:37:28,468-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:37:38,476-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:37:40,798-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:37:47,951-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:37:48,484-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:37:58,492-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:38:08,500-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:38:18,117-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:38:18,508-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:38:24,425-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:38:28,516-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:38:38,525-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:38:48,533-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:38:55,435-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:38:58,541-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:39:00,899-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:39:08,549-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:39:18,558-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:39:28,566-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:39:32,753-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:39:37,373-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:39:38,574-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:39:48,582-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:39:58,590-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:40:08,598-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 05:40:10,071-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:40:13,847-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:40:18,607-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:40:28,615-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:40:38,623-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:40:47,390-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:40:48,631-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:40:50,321-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:40:58,639-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:41:08,647-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:41:18,656-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:41:24,708-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:41:26,795-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:41:28,664-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:41:38,672-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:41:48,680-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:41:58,688-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:42:02,026-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:42:03,269-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:42:08,696-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:42:18,705-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:42:28,713-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:42:38,721-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:42:39,344-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:42:39,743-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:42:48,729-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:42:58,737-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:43:08,745-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 05:43:16,216-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:43:16,662-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:43:18,753-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:43:28,762-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:43:38,770-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:43:48,778-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:43:52,691-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:43:53,980-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:43:58,786-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:44:08,794-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:44:18,802-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:44:28,810-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:44:29,165-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:44:31,298-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:44:38,818-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:44:48,826-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:44:58,835-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:45:05,639-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:45:08,616-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:45:08,843-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:45:18,851-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:45:28,859-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:45:38,867-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 05:45:42,114-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 05:45:45,934-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 05:45:48,875-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:45:58,883-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:46:08,891-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 05:46:18,588-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 05:46:18,899-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:46:23,253-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:46:28,907-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:46:38,915-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:46:48,923-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:46:55,061-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:46:58,931-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:47:00,571-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:47:08,939-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:47:18,947-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:47:28,955-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:47:31,535-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:47:37,889-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:47:38,963-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:47:48,971-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:47:58,979-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:48:08,009-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:48:08,987-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:48:15,207-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:48:18,995-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:48:29,003-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:48:39,011-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:48:44,484-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:48:49,019-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 05:48:52,524-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 05:48:59,027-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:49:09,035-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:49:19,043-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:49:20,957-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:49:29,051-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:49:29,843-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:49:39,059-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:49:49,067-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 05:49:57,431-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 05:49:59,075-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:50:07,161-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:50:09,083-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:50:19,091-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:50:29,099-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:50:33,906-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:50:39,107-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:50:44,479-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:50:49,115-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:50:59,122-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:51:09,130-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:51:10,380-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:51:19,138-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:51:21,797-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:51:29,146-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:51:39,154-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:51:46,854-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:51:49,162-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:51:59,115-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:51:59,170-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:52:09,177-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:52:19,185-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:52:23,328-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:52:29,193-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:52:36,433-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:52:39,201-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:52:49,209-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:52:59,217-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:52:59,802-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:53:09,225-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:53:13,751-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:53:19,233-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:53:29,241-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:53:36,276-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:53:39,249-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:53:49,257-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:53:51,069-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:53:59,265-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:54:09,273-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:54:12,750-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:54:19,281-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:54:28,387-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:54:29,288-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:54:39,296-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 05:54:49,225-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 05:54:49,304-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:54:59,312-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:55:05,705-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:55:09,319-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:55:19,327-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 05:55:25,699-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 05:55:29,335-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:55:39,343-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:55:43,023-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:55:49,351-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:55:59,359-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:56:02,172-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:56:09,366-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:56:19,374-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 05:56:20,342-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 05:56:29,382-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:56:38,646-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:56:39,390-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:56:49,399-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 05:56:57,660-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 05:56:59,407-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:57:09,416-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:57:15,120-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:57:19,425-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:57:29,433-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:57:34,977-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:57:39,441-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:57:49,450-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 05:57:51,595-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 05:57:59,459-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:58:09,467-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:58:12,295-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:58:19,475-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:58:28,068-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:58:29,483-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:58:39,491-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:58:49,499-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 05:58:49,614-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 05:58:59,508-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-2013-11-07 05:59:04,544-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-No streams
-2013-11-07 05:59:09,517-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:59:19,529-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 05:59:26,931-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 05:59:29,539-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:59:39,549-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 05:59:41,016-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 05:59:49,560-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 05:59:59,570-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 06:00:04,250-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 06:00:09,581-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:00:17,490-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:00:19,591-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:00:29,602-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:00:39,612-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:00:41,567-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:00:49,626-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 06:00:53,965-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 06:00:59,638-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:01:09,649-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 06:01:18,886-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 06:01:19,660-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:01:29,680-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 06:01:30,439-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 06:01:39,692-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:01:49,696-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:01:56,203-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:01:59,715-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:02:06,913-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:02:09,731-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:02:19,746-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:02:29,758-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:02:33,522-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:02:39,766-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 06:02:43,388-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 06:02:49,776-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:02:59,784-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:03:09,796-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:03:10,839-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:03:19,807-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:03:19,862-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:03:29,817-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:03:39,826-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:03:48,157-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:03:49,836-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 06:03:56,336-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 06:03:59,846-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:04:09,854-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:04:19,863-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:04:25,476-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:04:29,872-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 06:04:32,811-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 06:04:39,882-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:04:49,891-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:04:59,899-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:05:02,795-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 06:05:09,284-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 06:05:09,910-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:05:19,919-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:05:29,927-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:05:39,940-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:05:40,113-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 06:05:45,758-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 06:05:49,952-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:05:59,961-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:06:09,972-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:06:17,431-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:06:19,985-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:06:22,232-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:06:29,994-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:06:40,005-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:06:50,015-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:06:54,749-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 06:06:58,706-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 06:07:00,024-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:07:10,033-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:07:20,043-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:07:30,063-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 06:07:32,068-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 06:07:35,181-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 06:07:40,073-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:07:50,076-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:08:00,079-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 06:08:09,386-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 06:08:10,091-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:08:11,654-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:08:20,103-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:08:30,111-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:08:40,120-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:08:46,703-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 06:08:48,129-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 06:08:50,128-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:09:00,138-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:09:10,148-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:09:20,156-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:09:24,021-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 06:09:24,602-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 06:09:30,166-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:09:40,177-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:09:50,188-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:10:00,200-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:10:01,077-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:10:01,339-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:10:10,209-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:10:20,218-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:10:30,228-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 06:10:37,551-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-07 06:10:38,658-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-No streams
-2013-11-07 06:10:40,239-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:10:50,248-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:11:00,257-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:11:10,266-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 06:11:14,024-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 06:11:15,976-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 06:11:20,276-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:11:30,285-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:11:40,294-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:11:50,306-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 06:11:50,498-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:11:53,293-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:12:00,315-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:12:10,325-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:12:20,334-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:12:26,973-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:12:30,344-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:12:30,611-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:12:40,353-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:12:50,361-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:13:00,369-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 06:13:03,448-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:13:07,929-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:13:10,378-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:13:20,386-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:13:30,394-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 06:13:39,922-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 06:13:40,401-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:13:45,248-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:13:50,410-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:14:00,419-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:14:10,428-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 06:14:16,396-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 06:14:20,436-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 06:14:22,567-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 06:14:30,447-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:14:40,462-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:14:50,471-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:14:52,870-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:14:59,884-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:15:00,479-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:15:10,487-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:15:20,496-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 06:15:29,345-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 06:15:30,504-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 06:15:37,203-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 06:15:40,513-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:15:50,522-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:16:00,530-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 06:16:05,818-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 06:16:10,538-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 06:16:14,520-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 06:16:20,546-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:16:30,554-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:16:40,563-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:16:42,293-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:16:50,572-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:16:51,838-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:17:00,580-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:17:10,588-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 06:17:18,767-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 06:17:20,596-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-07 06:17:29,157-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-No streams
-2013-11-07 06:17:30,605-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:17:40,614-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:17:50,622-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:17:55,241-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:18:00,630-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:18:06,474-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:18:10,638-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:18:20,646-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:18:30,654-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 06:18:31,715-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 06:18:40,662-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:18:43,792-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:18:50,670-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:19:00,680-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 06:19:08,189-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 06:19:10,689-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:19:20,698-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 06:19:21,111-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 06:19:30,707-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:19:40,718-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 06:19:44,663-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 06:19:50,728-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 06:19:58,429-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 06:20:00,739-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:20:10,747-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:20:20,756-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:20:21,138-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:20:30,765-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:20:35,746-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:20:40,775-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:20:50,785-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 06:20:57,611-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 06:21:00,793-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:21:10,802-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 06:21:13,065-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 06:21:20,810-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:21:30,818-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 06:21:34,086-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 06:21:40,827-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:21:50,382-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:21:50,835-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:22:00,843-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:22:10,559-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:22:10,852-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:22:20,862-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:22:27,701-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:22:30,871-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:22:40,880-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 06:22:47,034-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 06:22:50,889-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:23:00,899-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 06:23:05,020-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 06:23:10,909-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:23:20,917-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 06:23:23,507-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 06:23:30,927-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:23:40,937-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:23:42,338-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:23:50,946-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 06:23:59,983-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 06:24:00,955-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:24:10,964-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:24:19,656-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:24:20,973-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:24:30,983-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:24:36,455-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:24:40,992-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:24:51,001-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:24:56,974-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:25:01,010-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:25:11,020-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:25:12,929-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:25:21,029-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:25:31,038-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:25:34,293-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:25:41,047-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 06:25:49,404-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 06:25:51,056-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:26:01,065-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:26:11,074-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:26:11,611-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:26:21,083-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:26:25,877-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:26:31,092-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:26:41,101-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:26:48,929-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:26:51,110-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:27:01,118-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:27:02,351-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:27:11,127-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:27:21,136-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:27:26,248-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:27:31,146-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:27:38,826-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:27:41,157-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:27:51,165-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:28:01,173-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-07 06:28:03,566-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-No streams
-2013-11-07 06:28:11,182-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 06:28:15,299-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 06:28:21,190-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:28:31,199-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:28:40,883-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:28:41,207-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:28:51,215-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:28:51,773-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:29:01,223-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:29:11,231-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:29:18,201-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:29:21,239-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 06:29:28,246-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 06:29:31,247-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:29:41,255-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:29:51,263-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:29:55,519-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:30:01,271-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:30:04,721-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:30:11,278-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:30:21,286-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:30:31,294-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 06:30:32,838-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:30:41,195-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:30:41,302-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:30:51,310-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:31:01,318-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:31:10,155-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:31:11,326-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:31:17,669-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:31:21,334-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:31:31,342-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:31:41,351-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:31:47,473-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:31:51,358-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:31:54,143-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:32:01,367-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:32:11,375-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:32:21,383-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:32:24,791-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:32:30,617-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:32:31,391-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:32:41,399-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:32:51,407-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:33:01,415-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:33:02,109-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:33:07,091-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:33:11,423-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:33:21,431-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:33:31,440-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:33:39,428-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:33:41,448-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:33:43,565-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:33:51,455-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:34:01,463-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:34:11,471-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:34:16,746-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:34:20,039-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:34:21,479-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:34:31,487-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:34:41,495-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:34:51,502-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:34:54,064-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 06:34:56,513-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 06:35:01,510-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:35:11,517-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:35:21,526-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-07 06:35:31,383-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-No streams
-2013-11-07 06:35:31,534-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:35:32,987-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:35:41,542-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:35:51,551-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:36:01,558-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:36:08,700-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 06:36:09,461-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 06:36:11,566-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:36:21,575-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:36:31,583-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:36:41,591-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 06:36:45,935-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:36:46,018-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:36:51,598-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:37:01,606-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:37:11,614-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:37:21,622-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:37:22,409-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:37:23,336-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:37:31,630-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:37:41,638-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:37:51,646-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:37:58,883-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:38:00,654-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:38:01,653-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:38:11,661-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:38:21,669-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:38:31,677-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 06:38:35,357-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:38:37,972-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:38:41,685-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:38:51,693-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:39:01,701-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:39:11,708-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:39:11,831-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:39:15,290-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:39:21,716-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:39:31,724-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:39:41,732-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:39:48,305-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:39:51,740-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:39:52,608-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:40:01,748-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:40:11,756-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:40:21,763-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:40:24,779-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:40:29,926-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:40:31,771-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:40:41,779-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:40:51,787-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 06:41:01,253-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 06:41:01,795-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:41:07,244-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:41:11,803-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:41:21,810-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:41:31,818-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:41:37,727-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:41:41,826-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:41:44,562-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:41:51,834-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:42:01,842-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:42:11,849-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-07 06:42:14,201-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-No streams
-2013-11-07 06:42:21,857-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:42:21,881-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:42:31,865-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:42:41,873-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:42:50,676-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:42:51,881-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:42:59,199-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:43:01,889-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:43:11,896-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:43:21,904-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:43:27,150-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:43:31,912-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:43:36,517-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:43:41,920-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:43:51,927-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:44:01,935-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:44:03,624-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:44:11,943-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:44:13,835-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:44:21,951-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:44:31,959-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:44:40,098-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:44:41,966-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:44:51,153-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:44:51,974-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:45:01,982-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:45:11,990-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 6ms
-2013-11-07 06:45:16,577-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 6ms
-No streams
-2013-11-07 06:45:21,998-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:45:28,471-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:45:32,006-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:45:42,014-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:45:52,022-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 06:45:53,047-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 06:46:02,029-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:46:05,789-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:46:12,037-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:46:22,045-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:46:29,520-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:46:32,053-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:46:42,061-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:46:43,107-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:46:52,069-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:47:02,077-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:47:05,994-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:47:12,085-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-07 06:47:20,425-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-07 06:47:22,092-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:47:32,100-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:47:42,108-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:47:42,468-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:47:52,116-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:47:57,743-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:48:02,124-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:48:12,131-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 06:48:18,941-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 06:48:22,139-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:48:32,147-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:48:35,061-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:48:42,155-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:48:52,163-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:48:55,416-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:49:02,171-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:49:12,179-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:49:12,379-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:49:22,187-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:49:31,890-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:49:32,194-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:49:42,202-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:49:49,697-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:49:52,210-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:50:02,218-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:50:08,364-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:50:12,226-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:50:22,234-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-07 06:50:27,016-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-07 06:50:32,242-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:50:42,250-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:50:44,838-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:50:52,258-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:51:02,266-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:51:04,334-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:51:12,274-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:51:21,312-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:51:22,282-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:51:32,290-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-07 06:51:41,652-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-07 06:51:42,298-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:51:52,306-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:51:57,786-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:52:02,314-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:52:12,322-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:52:18,970-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:52:22,330-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:52:32,338-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:52:34,260-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:52:42,346-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:52:52,355-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:52:56,288-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:53:02,363-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 35ms
-2013-11-07 06:53:10,768-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 35ms
-No streams
-2013-11-07 06:53:12,371-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:53:22,379-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:53:32,387-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-07 06:53:33,606-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-No streams
-2013-11-07 06:53:42,395-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:53:47,208-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:53:52,403-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:54:02,411-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:54:10,925-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:54:12,420-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:54:22,428-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 06:54:23,681-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 06:54:32,436-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:54:42,444-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-07 06:54:48,243-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-07 06:54:52,451-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:55:00,156-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:55:02,459-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:55:12,467-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:55:22,475-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:55:25,561-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:55:32,482-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:55:36,630-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:55:42,490-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:55:52,498-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:56:02,505-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:56:02,879-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:56:12,513-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:56:13,104-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:56:22,520-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:56:32,527-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-07 06:56:40,197-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-07 06:56:42,535-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-07 06:56:49,577-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-No streams
-2013-11-07 06:56:52,543-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:57:02,551-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:57:12,559-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:57:17,515-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:57:22,568-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:57:26,052-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:57:32,576-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:57:42,584-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:57:52,592-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-07 06:57:54,833-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:58:02,526-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:58:02,600-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:58:12,608-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:58:22,616-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:58:32,151-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:58:32,624-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:58:39,000-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:58:42,633-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:58:52,641-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:59:02,648-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:59:09,469-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:59:12,656-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 06:59:15,474-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 06:59:22,664-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:59:32,672-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 06:59:42,680-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-07 06:59:46,787-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 06:59:51,949-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 06:59:52,688-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:00:02,696-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:00:12,704-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:00:22,712-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:00:24,106-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:00:28,422-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:00:32,720-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:00:42,728-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:00:52,736-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-07 07:01:01,424-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-07 07:01:02,744-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:01:04,896-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:01:12,752-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:01:22,760-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:01:32,768-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:01:38,742-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:01:41,370-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:01:42,776-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:01:52,784-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:02:02,791-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:02:12,799-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:02:16,060-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:02:17,844-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:02:22,807-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:02:32,815-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:02:42,823-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:02:52,831-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-07 07:02:53,378-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:02:54,318-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:03:02,839-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:03:12,847-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:03:22,855-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:03:30,696-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:03:30,792-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:03:32,863-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:03:42,871-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:03:52,879-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:04:02,887-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:04:07,266-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-07 07:04:08,014-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-07 07:04:12,894-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:04:22,902-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:04:32,910-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:04:42,918-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:04:43,740-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-07 07:04:45,331-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-No streams
-2013-11-07 07:04:52,926-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:05:02,934-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:05:12,942-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:05:20,214-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:05:22,650-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:05:22,950-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:05:32,958-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:05:42,966-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:05:52,974-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-07 07:05:56,689-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-07 07:05:59,968-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-07 07:06:02,981-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:06:12,989-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:06:22,997-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:06:33,005-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:06:33,162-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-07 07:06:37,285-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-No streams
-2013-11-07 07:06:43,013-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:06:53,021-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:07:03,029-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:07:09,636-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:07:13,037-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:07:14,604-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:07:23,044-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:07:33,052-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:07:43,060-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:07:46,110-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-07 07:07:51,922-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-07 07:07:53,068-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:08:03,076-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:08:13,084-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-07 07:08:22,584-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-No streams
-2013-11-07 07:08:23,092-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:08:29,240-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:08:33,100-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:08:43,107-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:08:53,115-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:08:59,058-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:09:03,123-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:09:06,558-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:09:13,131-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:09:23,139-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:09:33,147-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-07 07:09:35,531-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-No streams
-2013-11-07 07:09:43,154-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-07 07:09:43,876-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-07 07:09:53,162-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:10:03,170-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:10:12,006-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:10:13,178-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:10:21,194-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:10:23,185-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:10:33,193-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:10:43,201-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:10:48,480-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:10:53,209-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:10:58,512-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:11:03,216-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:11:13,224-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:11:23,232-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:11:24,954-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:11:33,240-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-07 07:11:35,830-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-07 07:11:43,247-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:11:53,255-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:12:01,428-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:12:03,263-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:12:13,148-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:12:13,271-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:12:23,278-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:12:33,286-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:12:37,902-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:12:43,294-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-07 07:12:50,466-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-07 07:12:53,302-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:13:03,310-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:13:13,317-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:13:14,376-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:13:23,325-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-07 07:13:27,784-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-No streams
-2013-11-07 07:13:33,333-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:13:43,340-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:13:50,850-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:13:53,348-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:14:03,356-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:14:05,102-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:14:13,364-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:14:23,372-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:14:27,324-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:14:33,380-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-07 07:14:42,420-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-07 07:14:43,388-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:14:53,396-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:15:03,404-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-07 07:15:03,799-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-No streams
-2013-11-07 07:15:13,413-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:15:19,738-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:15:23,421-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:15:33,429-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:15:40,272-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:15:43,437-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:15:53,445-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-07 07:15:57,056-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-07 07:16:03,452-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:16:13,460-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:16:16,746-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:16:23,468-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:16:33,476-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-07 07:16:34,374-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-No streams
-2013-11-07 07:16:43,484-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:16:53,220-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:16:53,492-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:17:03,500-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:17:11,693-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:17:13,508-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:17:23,516-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:17:29,694-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:17:33,524-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:17:43,532-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-07 07:17:49,011-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-07 07:17:53,540-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:18:03,547-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:18:06,168-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:18:13,555-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:18:23,563-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:18:26,329-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:18:33,571-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-07 07:18:42,643-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-No streams
-2013-11-07 07:18:43,579-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:18:53,587-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:19:03,595-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-07 07:19:03,647-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-07 07:19:13,603-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:19:19,116-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:19:23,611-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:19:33,619-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-07 07:19:40,965-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-No streams
-2013-11-07 07:19:43,626-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:19:53,634-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:19:55,590-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:20:03,642-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:20:13,650-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:20:18,284-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:20:23,658-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:20:32,064-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:20:33,666-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:20:43,674-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:20:53,682-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-07 07:20:55,602-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-07 07:21:03,691-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-07 07:21:08,539-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-No streams
-2013-11-07 07:21:13,699-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:21:23,707-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:21:32,920-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:21:33,714-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:21:43,722-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:21:45,012-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:21:53,730-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:22:03,738-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-07 07:22:10,238-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-07 07:22:13,746-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 2ms, rtt: 3ms
-2013-11-07 07:22:21,488-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 2ms, rtt: 3ms
-No streams
-2013-11-07 07:22:23,753-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:22:33,761-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:22:43,769-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-07 07:22:47,555-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-No streams
-2013-11-07 07:22:53,777-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:22:57,960-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:23:03,785-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:23:13,792-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:23:23,800-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:23:24,874-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:23:33,808-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:23:34,434-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:23:43,816-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:23:53,824-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-07 07:24:02,192-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-07 07:24:03,831-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:24:10,908-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:24:13,839-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:24:23,847-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:24:33,855-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:24:39,510-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:24:43,863-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:24:47,382-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:24:53,871-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:25:03,878-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:25:13,886-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-07 07:25:16,828-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:25:23,856-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:25:23,894-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:25:33,902-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:25:43,909-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:25:53,917-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-07 07:25:54,146-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:26:00,330-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:26:03,925-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:26:13,933-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:26:23,941-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:26:31,464-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:26:33,949-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:26:36,804-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:26:43,956-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:26:53,964-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:27:03,972-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-07 07:27:08,782-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:27:13,278-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:27:13,980-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:27:23,987-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:27:33,995-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:27:44,003-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:27:46,100-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:27:49,752-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:27:54,011-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:28:04,019-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:28:14,027-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-07 07:28:23,418-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-07 07:28:24,034-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:28:26,226-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:28:34,042-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:28:44,050-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:28:54,058-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:29:00,736-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-07 07:29:02,701-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-No streams
-2013-11-07 07:29:04,066-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:29:14,073-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:29:24,081-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:29:34,089-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-07 07:29:38,054-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:29:39,174-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:29:44,097-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:29:54,105-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:30:04,112-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:30:14,120-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-07 07:30:15,372-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-07 07:30:15,648-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-07 07:30:24,128-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:30:34,136-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:30:44,144-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-07 07:30:52,123-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:30:52,690-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:30:54,151-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:31:04,159-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:31:14,167-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:31:24,175-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:31:28,596-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-07 07:31:30,008-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-07 07:31:34,183-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:31:44,190-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:31:54,198-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:32:04,206-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:32:05,070-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-07 07:32:07,325-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-No streams
-2013-11-07 07:32:14,214-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:32:24,222-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:32:34,229-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-07 07:32:41,543-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-No streams
-2013-11-07 07:32:44,237-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:32:44,644-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:32:54,245-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:33:04,253-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:33:14,261-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:33:18,018-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-07 07:33:21,962-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-07 07:33:24,268-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:33:34,276-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:33:44,284-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:33:54,292-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-07 07:33:54,492-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:33:59,280-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:34:04,300-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:34:14,307-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:34:24,315-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:34:30,966-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:34:34,323-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-07 07:34:36,598-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-07 07:34:44,331-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:34:54,339-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:35:04,347-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:35:07,440-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-07 07:35:13,916-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-07 07:35:14,355-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:35:24,362-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:35:34,370-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-07 07:35:43,915-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-No streams
-2013-11-07 07:35:44,378-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:35:51,234-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:35:54,386-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:36:04,394-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:36:14,401-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:36:20,388-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:36:24,409-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-07 07:36:28,553-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-No streams
-2013-11-07 07:36:34,417-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:36:44,424-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:36:54,433-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:36:56,862-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:37:04,441-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:37:05,870-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:37:14,449-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:37:24,457-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:37:33,336-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:37:34,464-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-07 07:37:43,188-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-07 07:37:44,472-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:37:54,480-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:38:04,487-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:38:09,810-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:38:14,495-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:38:20,506-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:38:24,503-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:38:34,511-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:38:44,519-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:38:46,284-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:38:54,526-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-07 07:38:57,824-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-07 07:39:04,534-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:39:14,542-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:39:22,758-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:39:24,550-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:39:34,557-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:39:35,142-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:39:44,565-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:39:54,573-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:39:59,232-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:40:04,581-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-07 07:40:12,460-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-07 07:40:14,589-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:40:24,596-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:40:34,604-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-07 07:40:35,707-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-No streams
-2013-11-07 07:40:44,612-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-07 07:40:49,778-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-07 07:40:54,620-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:41:04,627-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-07 07:41:12,181-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-No streams
-2013-11-07 07:41:14,635-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:41:24,643-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:41:27,096-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:41:34,651-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:41:44,658-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:41:48,655-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:41:54,666-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-07 07:42:04,414-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-07 07:42:04,674-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:42:14,681-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:42:24,689-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-07 07:42:25,130-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-No streams
-2013-11-07 07:42:34,697-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-07 07:42:41,732-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-No streams
-2013-11-07 07:42:44,704-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:42:54,712-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:43:01,603-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:43:04,720-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:43:14,727-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:43:19,050-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:43:24,735-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:43:34,742-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:43:38,077-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:43:44,750-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:43:54,757-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-07 07:43:56,368-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-07 07:44:04,765-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:44:14,551-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:44:14,773-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:44:24,781-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:44:33,686-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:44:34,789-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:44:44,797-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:44:51,025-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:44:54,805-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:45:04,812-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-07 07:45:11,004-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-07 07:45:14,820-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:45:24,828-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:45:27,499-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:45:34,836-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:45:44,844-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-07 07:45:48,322-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-07 07:45:54,851-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-07 07:46:03,973-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-No streams
-2013-11-07 07:46:04,859-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:46:14,867-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:46:24,875-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:46:25,640-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:46:34,883-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:46:40,448-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:46:44,890-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:46:54,898-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-07 07:47:02,958-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-07 07:47:04,906-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:47:14,914-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-07 07:47:16,922-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-No streams
-2013-11-07 07:47:24,922-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:47:34,930-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:47:40,276-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:47:44,938-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:47:53,397-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:47:54,946-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:48:04,954-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:48:14,962-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-07 07:48:17,594-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-07 07:48:24,970-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-07 07:48:29,871-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-No streams
-2013-11-07 07:48:34,978-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:48:44,985-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-07 07:48:54,912-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-No streams
-2013-11-07 07:48:54,994-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:49:05,002-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:49:06,346-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:49:15,010-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:49:25,018-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:49:32,231-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:49:35,026-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:49:42,820-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:49:45,034-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:49:55,042-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:50:05,050-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-07 07:50:09,548-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-No streams
-2013-11-07 07:50:15,058-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:50:19,294-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:50:25,066-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:50:35,074-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:50:45,082-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:50:46,867-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:50:55,090-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:50:55,768-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:51:05,098-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:51:15,106-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-07 07:51:24,185-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-07 07:51:25,114-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 07:51:32,243-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 07:51:35,123-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:51:45,131-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:51:55,139-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:52:01,503-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:52:05,147-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:52:08,716-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:52:15,155-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:52:25,163-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:52:35,171-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:52:38,821-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:52:45,179-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-07 07:52:45,190-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-07 07:52:55,187-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:53:05,195-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:53:15,203-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-07 07:53:16,139-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:53:21,664-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:53:25,212-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:53:35,220-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:53:45,228-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:53:53,458-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:53:55,236-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:53:58,138-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:54:05,244-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:54:15,252-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:54:25,260-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-07 07:54:30,775-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:54:34,612-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:54:35,268-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:54:45,276-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:54:55,284-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:55:05,292-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:55:08,094-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-07 07:55:11,085-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-No streams
-2013-11-07 07:55:15,300-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:55:25,309-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:55:35,317-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:55:45,325-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-07 07:55:45,412-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:55:47,560-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:55:55,333-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:56:05,341-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:56:15,349-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:56:22,730-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-07 07:56:24,034-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-07 07:56:25,357-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:56:35,365-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:56:45,373-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:56:55,382-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-07 07:57:00,048-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:57:00,508-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:57:05,390-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:57:15,397-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:57:25,405-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:57:35,413-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:57:36,982-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-07 07:57:37,365-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-No streams
-2013-11-07 07:57:45,421-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:57:55,429-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:58:05,437-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:58:13,456-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:58:14,684-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:58:15,445-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:58:25,453-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:58:35,460-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:58:45,468-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:58:49,930-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-07 07:58:52,001-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-No streams
-2013-11-07 07:58:55,476-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:59:05,484-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:59:15,492-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:59:25,500-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:59:26,404-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 07:59:29,320-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 07:59:35,508-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:59:45,516-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 07:59:55,524-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:00:02,878-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:00:05,532-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-07 08:00:06,638-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-07 08:00:15,540-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:00:25,549-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:00:35,557-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:00:39,352-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:00:43,956-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:00:45,565-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:00:55,573-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:01:05,581-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:01:15,589-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:01:15,826-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:01:21,274-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:01:25,597-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:01:35,606-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:01:45,614-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:01:52,300-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:01:55,622-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-07 08:01:58,591-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-No streams
-2013-11-07 08:02:05,630-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:02:15,638-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:02:25,646-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:02:28,774-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:02:35,654-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:02:35,910-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:02:45,662-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:02:55,670-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-07 08:03:05,247-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-No streams
-2013-11-07 08:03:05,678-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-07 08:03:13,228-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-07 08:03:15,686-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:03:25,694-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:03:35,702-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:03:41,723-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:03:45,710-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:03:50,546-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:03:55,718-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:04:05,726-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:04:15,735-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:04:18,196-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:04:25,743-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:04:27,864-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:04:35,751-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:04:45,759-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:04:54,670-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:04:55,767-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:05:05,183-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:05:05,775-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:05:15,783-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:05:25,791-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:05:31,144-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:05:35,799-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:05:42,500-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:05:45,807-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:05:55,815-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:06:05,823-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:06:07,618-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:06:15,831-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-07 08:06:19,818-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-No streams
-2013-11-07 08:06:25,839-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:06:35,847-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 08:06:44,093-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 08:06:45,855-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:06:55,863-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:06:57,136-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:07:05,871-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:07:15,879-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:07:20,566-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:07:25,887-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-07 08:07:34,453-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-No streams
-2013-11-07 08:07:35,896-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:07:45,904-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:07:55,912-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:07:57,040-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:08:05,920-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:08:11,772-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:08:15,928-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:08:25,936-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:08:33,514-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:08:35,944-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:08:45,952-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-07 08:08:49,090-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-07 08:08:55,960-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:09:05,968-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:09:09,988-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:09:15,976-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:09:25,984-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:09:26,408-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:09:35,992-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:09:46,000-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:09:46,462-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:09:56,008-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:10:03,726-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:10:06,016-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:10:16,024-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:10:22,936-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:10:26,032-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:10:36,040-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-07 08:10:41,045-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-No streams
-2013-11-07 08:10:46,048-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:10:56,056-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:10:59,410-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:11:06,064-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:11:16,072-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:11:18,362-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:11:26,081-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:11:35,884-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:11:36,089-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:11:46,097-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-07 08:11:55,680-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-07 08:11:56,105-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:12:06,113-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:12:12,358-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:12:16,121-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:12:26,129-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:12:32,998-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:12:36,138-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:12:46,146-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:12:48,832-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:12:56,154-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:13:06,162-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:13:10,316-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:13:16,170-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:13:25,306-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:13:26,178-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:13:36,186-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:13:46,194-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-07 08:13:47,634-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-No streams
-2013-11-07 08:13:56,204-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 08:14:01,781-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 08:14:06,212-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:14:16,220-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:14:24,953-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:14:26,228-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:14:36,236-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:14:38,254-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:14:46,244-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:14:56,252-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:15:02,271-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:15:06,260-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 08:15:14,729-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 08:15:16,268-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:15:26,276-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:15:36,284-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-07 08:15:39,590-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-No streams
-2013-11-07 08:15:46,292-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:15:51,202-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:15:56,300-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:16:06,308-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:16:16,316-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:16:16,907-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:16:26,324-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:16:27,676-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:16:36,332-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:16:46,340-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:16:54,225-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:16:56,348-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:17:04,150-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:17:06,356-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:17:16,365-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:17:26,373-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:17:31,543-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:17:36,381-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:17:40,624-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:17:46,389-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:17:56,397-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:18:06,405-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:18:08,861-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:18:16,413-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:18:17,098-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:18:26,421-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:18:36,429-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:18:46,179-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:18:46,437-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:18:53,572-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:18:56,445-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:19:06,453-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:19:16,462-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:19:23,497-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:19:26,470-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:19:30,046-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:19:36,478-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:19:46,486-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:19:56,494-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-07 08:20:00,815-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-No streams
-2013-11-07 08:20:06,502-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:20:06,520-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:20:16,510-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:20:26,518-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:20:36,526-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:20:38,134-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 08:20:42,995-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 08:20:46,534-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:20:56,542-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:21:06,550-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:21:15,452-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:21:16,558-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 08:21:19,469-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 08:21:26,566-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:21:36,574-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:21:46,582-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:21:52,770-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:21:55,942-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:21:56,590-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:22:06,598-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:22:16,605-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:22:26,613-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:22:30,088-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:22:32,416-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:22:36,621-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:22:46,629-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:22:56,637-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:23:06,645-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 08:23:07,407-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:23:08,890-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:23:16,653-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:23:26,661-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:23:36,668-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:23:44,725-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:23:45,364-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:23:46,676-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:23:56,684-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:24:06,692-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:24:16,700-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:24:21,838-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:24:22,043-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:24:26,707-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:24:36,715-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:24:46,723-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:24:56,731-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:24:58,312-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:24:59,361-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:25:06,739-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:25:16,747-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:25:26,754-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:25:34,786-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:25:36,679-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:25:36,762-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:25:46,770-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:25:56,778-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:26:06,785-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:26:11,260-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:26:13,997-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:26:16,793-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:26:26,801-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:26:36,809-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:26:46,816-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:26:47,734-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 08:26:51,315-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 08:26:56,824-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:27:06,832-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:27:16,840-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:27:24,208-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:27:26,848-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:27:28,633-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:27:36,856-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:27:46,863-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:27:56,871-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:28:00,682-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:28:05,951-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:28:06,879-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:28:16,887-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:28:26,895-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:28:36,903-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:28:37,156-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:28:43,269-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:28:46,911-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:28:56,920-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:29:06,928-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:29:13,630-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:29:16,936-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:29:20,587-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:29:26,944-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:29:36,952-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:29:46,960-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 08:29:50,104-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 08:29:56,968-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:29:57,905-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:30:06,976-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:30:16,985-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:30:26,578-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:30:26,993-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:30:35,223-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:30:37,001-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:30:47,008-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:30:57,017-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:31:03,052-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:31:07,024-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 08:31:12,541-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 08:31:17,032-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:31:27,041-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:31:37,048-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 08:31:39,525-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 08:31:47,056-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:31:49,859-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:31:57,064-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:32:07,072-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:32:16,000-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:32:17,080-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:32:27,088-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 08:32:27,177-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 08:32:37,096-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:32:47,104-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:32:52,475-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:32:57,112-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:33:04,496-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:33:07,120-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:33:17,128-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:33:27,136-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:33:28,949-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:33:37,144-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 08:33:41,813-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 08:33:47,151-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:33:57,159-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 08:34:05,423-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 08:34:07,167-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:34:17,175-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:34:19,132-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:34:27,183-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:34:37,191-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:34:41,897-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:34:47,199-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:34:56,450-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:34:57,207-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:35:07,214-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:35:17,222-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 08:35:18,371-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 08:35:27,230-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 08:35:33,768-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 08:35:37,238-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:35:47,246-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:35:54,845-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:35:57,254-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:36:07,262-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:36:11,086-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:36:17,269-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:36:27,277-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 08:36:31,320-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 08:36:37,285-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:36:47,293-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 08:36:48,403-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 08:36:57,301-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:37:07,309-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:37:07,793-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:37:17,317-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:37:25,722-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:37:27,325-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:37:37,333-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 08:37:44,267-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 08:37:47,340-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:37:57,348-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:38:03,040-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:38:07,356-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:38:17,364-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:38:20,741-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:38:27,372-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:38:37,380-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 08:38:40,358-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 08:38:47,388-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 08:38:57,215-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 08:38:57,396-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:39:07,404-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:39:17,412-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:39:17,676-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:39:27,420-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:39:33,689-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:39:37,428-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:39:47,436-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 08:39:54,994-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 08:39:57,444-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:40:07,452-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:40:10,163-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:40:17,460-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:40:27,468-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:40:32,313-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:40:37,476-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 08:40:46,638-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 08:40:47,484-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:40:57,492-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:41:07,500-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 08:41:09,631-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 08:41:17,507-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:41:23,111-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:41:27,515-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:41:37,523-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:41:46,949-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:41:47,531-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:41:57,539-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 08:41:59,585-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 08:42:07,547-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:42:17,555-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 08:42:24,267-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 08:42:27,563-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 08:42:36,059-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 08:42:37,570-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:42:47,578-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:42:57,586-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:43:01,585-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:43:07,594-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-2013-11-07 08:43:12,535-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-No streams
-2013-11-07 08:43:17,602-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:43:27,609-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:43:37,617-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:43:38,903-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:43:47,625-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:43:49,007-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:43:57,633-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:44:07,641-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 08:44:16,221-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 08:44:17,649-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 08:44:25,481-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 08:44:27,657-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:44:37,664-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:44:47,672-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:44:53,539-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:44:57,680-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 08:45:01,955-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 08:45:07,688-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:45:17,696-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:45:27,704-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:45:30,858-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:45:37,713-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 08:45:38,428-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 08:45:47,721-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:45:57,729-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:46:07,737-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 08:46:08,176-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:46:14,903-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:46:17,745-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:46:27,753-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:46:37,761-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:46:45,494-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:46:47,769-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:46:51,377-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:46:57,777-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:47:07,785-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:47:17,793-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 08:47:22,812-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 08:47:27,801-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 08:47:27,851-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 08:47:37,809-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:47:47,817-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:47:57,824-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:48:00,130-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 08:48:04,325-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 08:48:07,832-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:48:17,840-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:48:27,848-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 08:48:37,448-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 08:48:37,856-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 08:48:40,799-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 08:48:47,864-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:48:57,872-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:49:07,880-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 08:49:14,767-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:49:17,273-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:49:17,888-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:49:27,896-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:49:37,904-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:49:47,912-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:49:52,084-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 08:49:53,747-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 08:49:57,920-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:50:07,928-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:50:17,935-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:50:27,943-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 08:50:29,403-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:50:30,221-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:50:37,951-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:50:47,959-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:50:57,967-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 08:51:06,695-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 08:51:06,720-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 08:51:07,975-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:51:17,983-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:51:27,991-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:51:37,999-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:51:43,169-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:51:44,038-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:51:48,007-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:51:58,015-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:52:08,023-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:52:18,031-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:52:19,643-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 08:52:21,356-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 08:52:28,039-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:52:38,047-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:52:48,055-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 08:52:56,117-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 08:52:58,063-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 08:52:58,674-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 08:53:08,073-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:53:18,081-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:53:28,089-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 08:53:32,592-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 08:53:35,992-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 08:53:38,097-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:53:48,105-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:53:58,113-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:54:08,121-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 08:54:09,066-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 08:54:13,310-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 08:54:18,129-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:54:28,137-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:54:38,145-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 08:54:45,540-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 08:54:48,153-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 08:54:50,628-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 08:54:58,161-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:55:08,169-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:55:18,177-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 08:55:22,014-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 08:55:27,946-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 08:55:28,185-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:55:38,192-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:55:48,200-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:55:58,208-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 08:55:58,488-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 08:56:05,264-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 08:56:08,216-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:56:18,223-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:56:28,231-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 08:56:34,961-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 08:56:38,239-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 08:56:42,581-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 08:56:48,246-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:56:58,254-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:57:08,261-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 08:57:11,436-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 08:57:18,269-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 08:57:19,900-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 08:57:28,277-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:57:38,285-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 08:57:47,910-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 08:57:48,292-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 08:57:57,218-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 08:57:58,300-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:58:08,308-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:58:18,316-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 08:58:24,384-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 08:58:28,324-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 08:58:34,536-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 08:58:38,332-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:58:48,340-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:58:58,347-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 08:59:00,859-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 08:59:08,355-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 08:59:11,854-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 08:59:18,363-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:59:28,371-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 08:59:37,332-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 08:59:38,379-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 08:59:48,387-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 08:59:49,173-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 08:59:58,395-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:00:08,403-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 4ms
-2013-11-07 09:00:13,809-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 4ms
-No streams
-2013-11-07 09:00:18,411-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:00:26,490-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:00:28,419-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:00:38,427-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:00:48,435-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:00:50,280-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:00:58,443-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:01:03,808-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:01:08,451-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:01:18,459-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:01:26,754-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:01:28,466-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:01:38,474-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:01:41,126-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:01:48,482-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:01:58,490-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:02:03,228-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:02:08,497-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:02:18,444-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:02:18,505-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:02:28,513-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:02:38,520-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 09:02:39,701-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 09:02:48,528-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 09:02:55,761-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 09:02:58,535-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:03:08,543-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:03:16,176-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:03:18,550-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:03:28,558-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:03:33,080-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:03:38,566-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:03:48,574-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 09:03:52,649-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 09:03:58,582-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:04:08,591-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:04:10,398-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:04:18,599-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:04:28,607-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:04:29,124-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:04:38,615-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:04:47,716-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:04:48,623-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:04:58,631-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:05:05,598-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:05:08,639-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:05:18,647-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:05:25,034-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:05:28,655-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:05:38,663-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 09:05:42,073-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 09:05:48,671-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:05:58,679-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:06:02,352-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:06:08,687-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:06:18,546-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:06:18,695-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:06:28,703-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:06:38,711-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:06:39,670-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:06:48,719-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:06:55,020-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:06:58,727-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:07:08,735-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:07:16,988-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:07:18,743-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:07:28,751-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:07:31,494-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:07:38,759-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:07:48,767-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:07:54,306-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:07:58,775-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:08:07,968-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:08:08,783-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:08:18,791-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:08:28,799-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:08:31,624-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:08:38,807-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:08:44,442-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:08:48,815-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:08:58,823-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:09:08,830-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:09:08,942-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:09:18,838-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:09:20,916-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:09:28,846-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:09:38,854-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:09:46,260-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:09:48,862-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:09:57,390-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:09:58,870-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:10:08,878-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:10:18,886-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:10:23,578-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:10:28,894-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 09:10:33,865-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 09:10:38,902-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:10:48,910-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:10:58,918-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-07 09:11:00,895-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-07 09:11:08,926-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 09:11:10,337-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 09:11:18,934-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:11:28,942-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:11:38,214-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:11:38,951-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:11:46,812-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:11:48,959-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:11:58,967-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:12:08,975-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:12:15,532-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:12:18,984-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:12:23,286-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:12:28,992-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:12:39,000-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:12:49,008-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:12:52,850-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:12:59,016-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:12:59,760-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:13:09,024-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:13:19,032-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:13:29,040-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:13:30,168-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 09:13:36,233-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 09:13:39,048-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:13:49,057-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:13:59,065-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 09:14:07,486-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 09:14:09,073-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:14:12,708-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:14:19,080-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:14:29,088-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:14:39,096-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:14:44,804-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:14:49,104-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:14:49,182-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:14:59,112-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:15:09,120-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:15:19,128-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:15:22,123-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:15:25,656-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:15:29,136-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:15:39,144-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:15:49,152-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:15:59,160-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:15:59,441-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:16:02,130-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:16:09,168-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:16:19,176-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:16:29,184-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:16:36,759-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:16:38,604-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:16:39,192-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:16:49,200-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:16:59,208-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:17:09,216-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 09:17:14,077-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:17:15,079-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:17:19,225-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:17:29,233-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:17:39,241-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:17:49,249-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:17:51,395-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:17:51,553-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:17:59,257-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:18:09,265-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:18:19,273-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:18:28,027-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:18:28,713-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:18:29,280-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:18:39,288-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:18:49,296-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:18:59,304-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:19:04,501-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:19:06,031-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:19:09,312-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:19:19,320-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:19:29,328-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:19:39,336-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:19:40,975-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:19:43,349-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:19:49,343-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:19:59,351-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:20:09,359-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:20:17,449-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:20:19,367-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 09:20:20,667-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 09:20:29,374-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:20:39,382-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:20:49,389-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 09:20:53,924-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:20:57,985-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:20:59,397-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:21:09,405-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:21:19,413-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:21:29,421-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:21:30,397-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-07 09:21:35,302-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-07 09:21:39,429-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:21:49,437-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:21:59,445-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 09:22:06,871-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 09:22:09,453-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:22:12,621-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:22:19,461-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:22:29,469-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:22:39,477-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:22:43,345-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:22:49,485-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:22:49,939-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:22:59,493-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:23:09,501-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:23:19,509-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 09:23:19,819-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 09:23:27,257-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 09:23:29,517-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:23:39,525-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:23:49,533-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:23:56,293-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:23:59,541-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:24:04,575-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:24:09,549-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:24:19,557-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:24:29,565-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:24:32,767-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:24:39,573-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 09:24:41,893-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 09:24:49,581-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:24:59,589-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:25:09,241-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:25:09,597-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:25:19,211-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:25:19,605-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:25:29,613-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:25:39,621-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:25:45,715-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:25:49,629-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:25:56,529-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:25:59,637-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:26:09,645-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:26:19,653-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:26:22,189-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:26:29,661-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 09:26:33,847-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 09:26:39,669-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:26:49,677-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:26:58,663-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:26:59,685-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:27:09,693-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:27:11,165-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:27:19,701-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:27:29,709-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:27:35,137-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:27:39,717-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:27:48,483-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:27:49,725-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:27:59,733-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:28:09,741-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:28:11,611-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:28:19,748-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 09:28:25,801-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 09:28:29,756-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:28:39,764-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:28:48,085-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:28:49,772-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:28:59,780-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:29:03,119-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:29:09,788-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:29:19,795-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:29:24,559-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:29:29,803-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:29:39,811-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-07 09:29:40,436-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-07 09:29:49,819-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:29:59,827-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:30:01,033-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:30:09,835-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 09:30:17,755-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 09:30:19,843-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:30:29,851-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:30:37,507-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:30:39,859-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:30:49,867-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:30:55,073-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:30:59,875-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:31:09,883-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 09:31:13,981-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 09:31:19,891-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:31:29,899-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:31:32,391-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:31:39,907-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:31:49,915-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:31:50,455-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:31:59,923-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 09:32:09,709-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 09:32:09,932-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:32:19,940-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-07 09:32:26,929-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-07 09:32:29,948-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:32:39,956-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 09:32:47,027-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 09:32:49,964-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:32:59,972-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:33:03,404-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:33:09,980-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:33:19,988-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:33:24,345-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:33:29,996-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:33:39,878-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:33:40,005-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:33:50,013-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:34:00,021-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 09:34:01,663-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 09:34:10,029-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:34:16,352-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:34:20,037-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:34:30,045-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 09:34:38,981-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 09:34:40,053-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:34:50,061-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 09:34:52,827-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 09:35:00,069-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:35:10,077-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 09:35:16,299-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 09:35:20,085-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:35:29,300-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:35:30,094-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:35:40,102-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:35:50,110-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:35:53,617-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:36:00,118-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:36:05,774-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:36:10,126-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:36:20,134-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:36:30,142-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 09:36:30,935-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 09:36:40,150-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-07 09:36:42,248-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-07 09:36:50,158-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:37:00,166-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 09:37:08,253-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 09:37:10,174-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 09:37:18,722-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 09:37:20,182-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:37:30,189-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:37:40,198-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-07 09:37:45,570-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-07 09:37:50,205-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:37:55,196-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:38:00,214-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:38:10,222-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:38:20,229-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 09:38:22,889-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 09:38:30,237-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-07 09:38:31,670-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-07 09:38:40,245-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:38:50,253-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 09:39:00,207-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 09:39:00,261-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 09:39:08,146-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 09:39:10,270-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:39:20,278-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:39:30,286-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:39:37,525-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:39:40,293-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:39:44,619-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:39:50,301-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:40:00,309-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:40:10,317-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 09:40:14,843-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 09:40:20,325-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 09:40:21,093-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 09:40:30,334-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:40:40,342-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:40:50,350-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 09:40:52,161-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:40:57,567-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:41:00,358-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:41:10,367-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:41:20,375-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:41:29,479-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:41:30,383-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 09:41:34,042-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 09:41:40,391-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:41:50,398-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:42:00,406-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 09:42:06,797-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 09:42:10,413-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:42:10,515-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:42:20,420-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:42:30,429-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:42:40,437-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:42:44,115-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:42:46,989-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:42:50,444-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:43:00,452-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:43:10,460-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:43:20,468-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 09:43:21,433-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:43:23,463-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:43:30,476-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:43:40,484-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:43:50,491-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 09:43:58,751-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 09:43:59,937-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 09:44:00,499-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:44:10,507-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:44:20,515-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:44:30,522-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 09:44:36,069-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-07 09:44:36,410-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-07 09:44:40,530-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:44:50,538-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:45:00,546-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:45:10,553-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:45:12,885-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 09:45:13,387-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 09:45:20,561-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:45:30,568-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:45:40,576-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 09:45:49,360-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 09:45:50,584-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-07 09:45:50,704-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-07 09:46:00,591-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:46:10,599-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:46:20,607-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:46:25,833-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 09:46:28,023-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 09:46:30,615-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:46:40,622-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:46:50,630-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:47:00,638-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:47:02,307-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:47:05,341-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:47:10,646-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:47:20,653-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:47:30,661-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:47:38,781-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:47:40,669-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 09:47:42,659-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 09:47:50,677-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:48:00,684-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:48:10,692-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 09:48:15,255-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 09:48:19,977-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 09:48:20,700-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:48:30,708-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:48:40,715-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:48:50,723-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-07 09:48:51,728-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 09:48:57,295-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 09:49:00,731-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:49:10,739-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:49:20,746-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 09:49:28,203-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 09:49:30,754-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 09:49:34,613-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 09:49:40,762-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:49:50,770-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:50:00,778-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-07 09:50:04,677-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-07 09:50:10,785-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-07 09:50:11,931-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-07 09:50:20,793-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:50:30,801-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:50:40,809-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 09:50:41,152-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 09:50:49,249-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 09:50:50,817-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:51:00,825-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:51:10,832-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 09:51:17,626-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 09:51:20,840-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-07 09:51:26,567-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-07 09:51:30,848-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:51:40,856-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:51:50,864-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 09:51:54,100-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 09:52:00,871-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 09:52:03,885-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 09:52:10,879-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:52:20,887-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 09:52:30,574-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 09:52:30,895-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:52:40,903-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 09:52:41,203-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 09:52:50,911-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:53:00,919-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:53:07,048-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:53:10,926-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 09:53:18,521-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 09:53:20,934-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:53:30,942-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:53:40,950-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 09:53:43,522-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 09:53:50,957-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:53:55,839-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:54:00,965-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:54:10,973-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:54:19,996-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:54:20,981-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:54:30,988-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 09:54:33,157-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 09:54:40,996-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:54:51,004-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-2013-11-07 09:54:56,472-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-No streams
-2013-11-07 09:55:01,012-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 09:55:10,475-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 09:55:11,020-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:55:21,027-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:55:31,035-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:55:32,944-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:55:41,043-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 09:55:47,793-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 09:55:51,051-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:56:01,059-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:56:09,418-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:56:11,067-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:56:21,074-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 09:56:25,111-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 09:56:31,082-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:56:41,090-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:56:45,893-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:56:51,098-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:57:01,105-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-07 09:57:02,428-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-07 09:57:11,113-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:57:21,121-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:57:22,367-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:57:31,129-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 09:57:39,747-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 09:57:41,136-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:57:51,144-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 09:57:58,842-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 09:58:01,152-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:58:11,160-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:58:17,065-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:58:21,168-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:58:31,176-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:58:35,315-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:58:41,183-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:58:51,191-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 09:58:54,383-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 09:59:01,199-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:59:11,207-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 09:59:11,790-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 09:59:21,215-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 09:59:31,223-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 09:59:31,701-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 09:59:41,231-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 09:59:48,263-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 09:59:51,239-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:00:01,247-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-07 10:00:09,019-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-07 10:00:11,255-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:00:21,263-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 10:00:24,738-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 10:00:31,271-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:00:41,279-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 10:00:46,338-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 10:00:51,287-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 10:01:01,211-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 10:01:01,295-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:01:11,303-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:01:21,310-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:01:23,656-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:01:31,317-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 10:01:37,685-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 10:01:41,325-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:01:51,333-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 10:02:00,974-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 10:02:01,341-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:02:11,349-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:02:14,159-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:02:21,357-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:02:31,365-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:02:38,292-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:02:41,373-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 10:02:50,633-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 10:02:51,381-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:03:01,389-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:03:11,397-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 10:03:15,610-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 10:03:21,405-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:03:27,107-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:03:31,412-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:03:41,420-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:03:51,428-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 10:03:52,928-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 10:04:01,436-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:04:03,581-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:04:11,444-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:04:21,452-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:04:30,247-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:04:31,460-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 10:04:40,056-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 10:04:41,468-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:04:51,476-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:05:01,484-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 10:05:07,565-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 10:05:11,492-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 10:05:16,530-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 10:05:21,500-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:05:31,508-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:05:41,516-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:05:44,883-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:05:51,524-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:05:53,003-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:06:01,532-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:06:11,540-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:06:21,548-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 10:06:22,201-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 10:06:29,477-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 10:06:31,556-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:06:41,564-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:06:51,572-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 10:06:59,519-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 10:07:01,580-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 10:07:05,951-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 10:07:11,588-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:07:21,596-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:07:31,604-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:07:36,837-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:07:41,612-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:07:42,425-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:07:51,620-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:08:01,628-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:08:11,636-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 10:08:14,156-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:08:18,899-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:08:21,644-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:08:31,652-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:08:41,660-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 10:08:51,473-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 10:08:51,668-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:08:55,373-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:09:01,676-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:09:11,684-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:09:21,692-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:09:28,791-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:09:31,700-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:09:31,847-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:09:41,708-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:09:51,715-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:10:01,723-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 10:10:06,109-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:10:08,321-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:10:11,731-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:10:21,739-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:10:31,747-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:10:41,755-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 10:10:43,427-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 44ms
-2013-11-07 10:10:44,838-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 44ms
-No streams
-2013-11-07 10:10:51,763-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:11:01,771-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:11:11,778-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:11:20,746-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:11:21,269-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:11:21,786-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:11:31,794-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:11:41,802-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:11:51,810-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:11:57,743-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-07 10:11:58,064-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-07 10:12:01,818-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:12:11,825-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:12:21,833-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:12:31,841-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:12:34,217-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:12:35,383-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:12:41,849-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:12:51,857-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:13:01,864-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 10:13:10,691-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 10:13:11,872-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-07 10:13:12,700-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-07 10:13:21,880-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:13:31,888-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:13:41,896-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:13:47,165-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 10:13:50,019-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 10:13:51,903-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:14:01,911-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:14:11,919-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:14:21,927-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:14:23,639-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:14:27,337-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:14:31,935-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:14:41,943-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:14:51,951-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:15:00,113-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:15:01,959-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 10:15:04,656-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 10:15:11,967-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:15:21,975-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:15:31,983-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 10:15:36,588-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:15:41,973-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:15:41,991-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:15:51,999-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:16:02,007-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:16:12,015-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:16:13,061-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-07 10:16:19,291-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-07 10:16:22,023-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:16:32,030-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:16:42,038-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 10:16:49,536-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 10:16:52,046-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:16:56,609-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:17:02,054-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:17:12,062-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:17:22,070-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 10:17:26,010-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 10:17:32,078-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:17:33,927-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:17:42,086-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:17:52,094-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:18:02,101-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:18:02,483-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 10:18:11,245-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 10:18:12,109-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:18:22,117-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:18:32,125-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-07 10:18:38,956-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-07 10:18:42,133-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:18:48,563-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:18:52,141-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:19:02,148-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:19:12,156-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:19:15,431-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:19:22,164-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 10:19:25,881-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 10:19:32,172-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:19:42,180-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:19:51,905-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:19:52,188-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:20:02,195-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:20:03,199-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:20:12,203-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:20:22,211-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:20:28,379-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:20:32,219-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:20:40,517-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:20:42,227-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:20:52,235-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:21:02,242-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:21:04,854-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:21:12,251-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:21:17,835-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:21:22,259-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:21:32,267-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:21:41,328-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:21:42,275-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:21:52,283-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:21:55,153-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:22:02,291-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:22:12,299-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 10:22:17,802-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 10:22:22,307-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:22:32,315-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 10:22:32,471-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 10:22:42,323-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:22:52,331-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:22:54,276-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:23:02,340-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:23:09,789-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:23:12,347-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:23:22,356-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:23:30,750-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:23:32,364-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:23:42,372-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 10:23:47,107-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 10:23:52,380-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:24:02,388-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 10:24:07,225-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 10:24:12,395-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:24:22,403-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:24:24,425-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:24:32,411-0600 INFO Multiplexer No streams
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 10:24:33,539-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50621] for addition
-2013-11-07 10:24:33,539-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50621] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:50621]
-2013-11-07 10:24:33,539-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:50621]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f78
-2013-11-07 10:24:33,576-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f78
-MetaChannel [context: cps-50621, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50621]
-2013-11-07 10:24:33,576-0600 INFO MetaChannel MetaChannel [context: cps-50621, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50621]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-07 10:24:33,576-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-07 10:24:33,616-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-07 10:24:33,616-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-07 10:24:33,697-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-07 10:24:33,743-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 93.0) submitting urn:job-0-1383759274855-1383759274856: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 10:24:33,743-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 93.0) submitting urn:job-0-1383759274855-1383759274856: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 10:24:33,744-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-07 10:24:33,744-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-07 10:24:33,749-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7518, JOBSTATUS) unregistering (send)
-2013-11-07 10:24:33,749-0600 INFO RequestHandler Handler(tag: 7518, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 10:24:33,756-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50622] for addition
-2013-11-07 10:24:33,756-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50622] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:50622]
-2013-11-07 10:24:33,756-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:50622]
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f77
-2013-11-07 10:24:33,777-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f77
-MetaChannel [context: cps-50622, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50622]
-2013-11-07 10:24:33,778-0600 INFO MetaChannel MetaChannel [context: cps-50622, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50622]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-07 10:24:33,778-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-07 10:24:33,818-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-07 10:24:33,899-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-07 10:24:33,944-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-crush.mcs.anl.gov:0 (quality: 31.0) submitting urn:job-1-1383759274858-1383759274859: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 10:24:33,944-0600 INFO Cpu crush.mcs.anl.gov:0 (quality: 31.0) submitting urn:job-1-1383759274858-1383759274859: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 10:24:33,949-0600 DEBUG Cpu Active workerid=crush.mcs.anl.gov:000000
-Handler(tag: 93, JOBSTATUS) unregistering (send)
-2013-11-07 10:24:33,949-0600 INFO RequestHandler Handler(tag: 93, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 10:24:33,955-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50623] for addition
-2013-11-07 10:24:33,955-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50623] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:50623]
-2013-11-07 10:24:33,955-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:50623]
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f76
-2013-11-07 10:24:33,979-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f76
-MetaChannel [context: cps-50623, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50623]
-2013-11-07 10:24:33,979-0600 INFO MetaChannel MetaChannel [context: cps-50623, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50623]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-07 10:24:33,979-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-07 10:24:34,020-0600 INFO RequestHandler Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-07 10:24:34,100-0600 INFO RequestHandler Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-07 10:24:35,832-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 10:24:35,832-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 10:24:35,832-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:24:35,832-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 7519, JOBSTATUS) unregistering (send)
-2013-11-07 10:24:35,832-0600 INFO RequestHandler Handler(tag: 7519, JOBSTATUS) unregistering (send)
-2013-11-07 10:24:35,832-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-Channel config: null -> {keepalive=null}
-2013-11-07 10:24:35,832-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50621] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:24:35,833-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50621] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-grind.mcs.anl.gov:0 (quality: 94.0) submitting urn:job-2-1383759274861-1383759274862: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 10:24:35,833-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 94.0) submitting urn:job-2-1383759274861-1383759274862: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-null: Channel shut down
-2013-11-07 10:24:35,833-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-50621, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50621]] binding to NullChannel
-2013-11-07 10:24:35,833-0600 INFO MetaChannel MetaChannel [context: cps-50621, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50621]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50621]
-2013-11-07 10:24:35,833-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50621]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:24:35,833-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 10:24:35,833-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:24:35,833-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 10:24:35,833-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:24:35,833-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 10:24:35,834-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:24:35,834-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 10:24:35,834-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:24:35,834-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 10:24:35,834-0600 INFO ChannelManager Channel exception handled
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:24:35,834-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel config: null -> {keepalive=null}
-2013-11-07 10:24:35,834-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50623] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:24:35,834-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50623] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-07 10:24:35,834-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-50623, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50623]] binding to NullChannel
-2013-11-07 10:24:35,834-0600 INFO MetaChannel MetaChannel [context: cps-50623, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50623]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50623]
-2013-11-07 10:24:35,834-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50623]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:24:35,834-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 10:24:35,835-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:24:35,835-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 10:24:35,835-0600 INFO ChannelManager Channel exception handled
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 10:24:35,838-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 10:24:35,838-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 10:24:35,839-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7520, JOBSTATUS) unregistering (send)
-2013-11-07 10:24:35,839-0600 INFO RequestHandler Handler(tag: 7520, JOBSTATUS) unregistering (send)
-Channel is closed. Removing.
-2013-11-07 10:24:35,845-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 10:24:35,845-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-07 10:24:35,846-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50623] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 10:24:35,846-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50623] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 10:24:35,846-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50623]
-2013-11-07 10:24:35,846-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50623]
-Channel is closed. Removing.
-2013-11-07 10:24:35,846-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 10:24:35,846-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-07 10:24:35,846-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50621] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 10:24:35,846-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50621] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 10:24:35,846-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50621]
-2013-11-07 10:24:35,846-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50621]
-2013-11-07 10:24:36,244-0600 DEBUG Cpu Completed
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:24:36,244-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-crush.mcs.anl.gov:0 jobTerminated
-2013-11-07 10:24:36,244-0600 INFO Cpu crush.mcs.anl.gov:0 jobTerminated
-Channel config: null -> {keepalive=null}
-2013-11-07 10:24:36,244-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-07 10:24:36,245-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-TCPChannel [type: server, contact: 127.0.0.1:50622] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:24:36,245-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50622] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 94, JOBSTATUS) unregistering (send)
-2013-11-07 10:24:36,245-0600 INFO RequestHandler Handler(tag: 94, JOBSTATUS) unregistering (send)
-2013-11-07 10:24:36,245-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-null: Channel shut down
-2013-11-07 10:24:36,245-0600 INFO AbstractTCPChannel null: Channel shut down
-2013-11-07 10:24:36,245-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-MetaChannel [context: cps-50622, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50622]] binding to NullChannel
-2013-11-07 10:24:36,245-0600 INFO MetaChannel MetaChannel [context: cps-50622, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50622]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50622]
-2013-11-07 10:24:36,245-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50622]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:24:36,245-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 10:24:36,245-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:24:36,245-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 10:24:36,245-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:24:36,245-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 10:24:36,245-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:24:36,245-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 10:24:36,246-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:24:36,246-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 10:24:36,246-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-07 10:24:36,247-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 10:24:36,247-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-07 10:24:36,247-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50622] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 10:24:36,247-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50622] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 10:24:36,247-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50622]
-2013-11-07 10:24:36,247-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50622]
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 10:24:37,852-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 10:24:37,852-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 10:24:37,852-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 10:24:37,852-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-07 10:24:37,852-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7521, JOBSTATUS) unregistering (send)
-2013-11-07 10:24:37,852-0600 INFO RequestHandler Handler(tag: 7521, JOBSTATUS) unregistering (send)
-2013-11-07 10:24:37,852-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-No streams
-2013-11-07 10:24:42,424-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:24:43,698-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:24:52,432-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:25:01,744-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:25:02,439-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:25:12,447-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 10:25:20,172-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 10:25:22,454-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:25:32,462-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:25:39,062-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:25:42,469-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:25:52,477-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:25:56,646-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 10:25:57,778-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50624] for addition
-2013-11-07 10:25:57,778-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50624] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:50624]
-2013-11-07 10:25:57,778-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:50624]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f75
-2013-11-07 10:25:57,813-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f75
-MetaChannel [context: cps-50624, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50624]
-2013-11-07 10:25:57,813-0600 INFO MetaChannel MetaChannel [context: cps-50624, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50624]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-07 10:25:57,813-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-07 10:25:57,853-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-07 10:25:57,853-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-07 10:25:57,934-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-07 10:25:57,941-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 95.0) submitting urn:job-0-1383759274864-1383759274865: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 10:25:57,941-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 95.0) submitting urn:job-0-1383759274864-1383759274865: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 10:25:57,942-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-07 10:25:57,942-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-07 10:25:57,946-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7522, JOBSTATUS) unregistering (send)
-2013-11-07 10:25:57,947-0600 INFO RequestHandler Handler(tag: 7522, JOBSTATUS) unregistering (send)
-2013-11-07 10:26:00,093-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 10:26:00,093-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 10:26:00,093-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:26:00,093-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 7523, JOBSTATUS) unregistering (send)
-2013-11-07 10:26:00,093-0600 INFO RequestHandler Handler(tag: 7523, JOBSTATUS) unregistering (send)
-2013-11-07 10:26:00,093-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Channel config: null -> {keepalive=null}
-2013-11-07 10:26:00,093-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-07 10:26:00,093-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-TCPChannel [type: server, contact: 127.0.0.1:50624] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:26:00,093-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50624] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-07 10:26:00,094-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-50624, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50624]] binding to NullChannel
-2013-11-07 10:26:00,094-0600 INFO MetaChannel MetaChannel [context: cps-50624, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50624]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50624]
-2013-11-07 10:26:00,094-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50624]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:26:00,094-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 10:26:00,094-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:26:00,094-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 10:26:00,094-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:26:00,094-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 10:26:00,094-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:26:00,094-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 10:26:00,094-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:26:00,094-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 10:26:00,094-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-07 10:26:00,100-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 10:26:00,100-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-07 10:26:00,100-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50624] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 10:26:00,100-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50624] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 10:26:00,100-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50624]
-2013-11-07 10:26:00,100-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50624]
-No streams
-2013-11-07 10:26:02,487-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:26:12,494-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 10:26:16,380-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 10:26:22,501-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:26:32,508-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 10:26:33,121-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102433.697, now=131107-102634.539, channel=TCPChannel [type: server, contact: 127.0.0.1:50621]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:26:34,539-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102433.697, now=131107-102634.539, channel=TCPChannel [type: server, contact: 127.0.0.1:50621]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 10:26:34,539-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102433.697, now=131107-102634.539, channel=TCPChannel [type: server, contact: 127.0.0.1:50621]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:26:34,539-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102433.697, now=131107-102634.539, channel=TCPChannel [type: server, contact: 127.0.0.1:50621]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 10:26:34,539-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102433.697, now=131107-102634.539, channel=TCPChannel [type: server, contact: 127.0.0.1:50621]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:26:34,539-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102433.697, now=131107-102634.539, channel=TCPChannel [type: server, contact: 127.0.0.1:50621]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 10:26:34,539-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102433.697, now=131107-102634.539, channel=TCPChannel [type: server, contact: 127.0.0.1:50621]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:26:34,539-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102433.697, now=131107-102634.539, channel=TCPChannel [type: server, contact: 127.0.0.1:50621]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 10:26:34,539-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102433.697, now=131107-102634.539, channel=TCPChannel [type: server, contact: 127.0.0.1:50621]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:26:34,539-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102433.697, now=131107-102634.539, channel=TCPChannel [type: server, contact: 127.0.0.1:50621]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-07 10:26:34,539-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50621] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102433.697, now=131107-102634.539, channel=TCPChannel [type: server, contact: 127.0.0.1:50621]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:26:34,539-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50621] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102433.697, now=131107-102634.539, channel=TCPChannel [type: server, contact: 127.0.0.1:50621]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 10:26:34,540-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102433.898, now=131107-102634.756, channel=TCPChannel [type: server, contact: 127.0.0.1:50622]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:26:34,756-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102433.898, now=131107-102634.756, channel=TCPChannel [type: server, contact: 127.0.0.1:50622]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 10:26:34,756-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102433.898, now=131107-102634.756, channel=TCPChannel [type: server, contact: 127.0.0.1:50622]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:26:34,756-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102433.898, now=131107-102634.756, channel=TCPChannel [type: server, contact: 127.0.0.1:50622]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 10:26:34,756-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102433.898, now=131107-102634.756, channel=TCPChannel [type: server, contact: 127.0.0.1:50622]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:26:34,756-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102433.898, now=131107-102634.756, channel=TCPChannel [type: server, contact: 127.0.0.1:50622]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 10:26:34,756-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102433.898, now=131107-102634.756, channel=TCPChannel [type: server, contact: 127.0.0.1:50622]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:26:34,756-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102433.898, now=131107-102634.756, channel=TCPChannel [type: server, contact: 127.0.0.1:50622]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 10:26:34,756-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102433.898, now=131107-102634.756, channel=TCPChannel [type: server, contact: 127.0.0.1:50622]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:26:34,756-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102433.898, now=131107-102634.756, channel=TCPChannel [type: server, contact: 127.0.0.1:50622]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-07 10:26:34,756-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50622] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102433.898, now=131107-102634.756, channel=TCPChannel [type: server, contact: 127.0.0.1:50622]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:26:34,756-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50622] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102433.898, now=131107-102634.756, channel=TCPChannel [type: server, contact: 127.0.0.1:50622]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 10:26:34,756-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102434.100, now=131107-102634.955, channel=TCPChannel [type: server, contact: 127.0.0.1:50623]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:26:34,955-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102434.100, now=131107-102634.955, channel=TCPChannel [type: server, contact: 127.0.0.1:50623]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 10:26:34,956-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102434.100, now=131107-102634.955, channel=TCPChannel [type: server, contact: 127.0.0.1:50623]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:26:34,956-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102434.100, now=131107-102634.955, channel=TCPChannel [type: server, contact: 127.0.0.1:50623]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-07 10:26:34,956-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50623] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102434.100, now=131107-102634.955, channel=TCPChannel [type: server, contact: 127.0.0.1:50623]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:26:34,956-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50623] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102434.100, now=131107-102634.955, channel=TCPChannel [type: server, contact: 127.0.0.1:50623]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 10:26:34,956-0600 INFO ChannelManager Channel already shut down
-No streams
-2013-11-07 10:26:42,515-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:26:52,523-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:26:53,698-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 10:26:55,624-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50625] for addition
-2013-11-07 10:26:55,624-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50625] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:50625]
-2013-11-07 10:26:55,624-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:50625]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f74
-2013-11-07 10:26:55,652-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f74
-MetaChannel [context: cps-50625, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50625]
-2013-11-07 10:26:55,653-0600 INFO MetaChannel MetaChannel [context: cps-50625, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50625]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-07 10:26:55,653-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-07 10:26:55,693-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-07 10:26:55,693-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-07 10:26:55,774-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-07 10:26:55,806-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 96.0) submitting urn:job-0-1383759274867-1383759274868: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 10:26:55,806-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 96.0) submitting urn:job-0-1383759274867-1383759274868: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 10:26:55,806-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-07 10:26:55,806-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-07 10:26:55,812-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7524, JOBSTATUS) unregistering (send)
-2013-11-07 10:26:55,813-0600 INFO RequestHandler Handler(tag: 7524, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 10:26:55,832-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50626] for addition
-2013-11-07 10:26:55,832-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50626] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:50626]
-2013-11-07 10:26:55,832-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:50626]
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f73
-2013-11-07 10:26:55,854-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f73
-MetaChannel [context: cps-50626, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50626]
-2013-11-07 10:26:55,854-0600 INFO MetaChannel MetaChannel [context: cps-50626, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50626]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-07 10:26:55,854-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-07 10:26:55,895-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-07 10:26:55,975-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-07 10:26:56,007-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-crush.mcs.anl.gov:0 (quality: 32.0) submitting urn:job-1-1383759274870-1383759274871: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 10:26:56,007-0600 INFO Cpu crush.mcs.anl.gov:0 (quality: 32.0) submitting urn:job-1-1383759274870-1383759274871: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 10:26:56,013-0600 DEBUG Cpu Active workerid=crush.mcs.anl.gov:000000
-Handler(tag: 95, JOBSTATUS) unregistering (send)
-2013-11-07 10:26:56,014-0600 INFO RequestHandler Handler(tag: 95, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 10:26:56,035-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50627] for addition
-2013-11-07 10:26:56,036-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50627] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:50627]
-2013-11-07 10:26:56,036-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:50627]
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f72
-2013-11-07 10:26:56,077-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f72
-MetaChannel [context: cps-50627, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50627]
-2013-11-07 10:26:56,077-0600 INFO MetaChannel MetaChannel [context: cps-50627, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50627]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-07 10:26:56,077-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-07 10:26:56,097-0600 INFO RequestHandler Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-07 10:26:56,178-0600 INFO RequestHandler Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-07 10:26:57,835-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 10:26:57,835-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 10:26:57,835-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:26:57,835-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 7525, JOBSTATUS) unregistering (send)
-2013-11-07 10:26:57,835-0600 INFO RequestHandler Handler(tag: 7525, JOBSTATUS) unregistering (send)
-2013-11-07 10:26:57,835-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-Channel config: null -> {keepalive=null}
-2013-11-07 10:26:57,835-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50625] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:26:57,835-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50625] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-grind.mcs.anl.gov:0 (quality: 97.0) submitting urn:job-2-1383759274873-1383759274874: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 10:26:57,835-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 97.0) submitting urn:job-2-1383759274873-1383759274874: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-null: Channel shut down
-2013-11-07 10:26:57,835-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-50625, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50625]] binding to NullChannel
-2013-11-07 10:26:57,835-0600 INFO MetaChannel MetaChannel [context: cps-50625, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50625]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50625]
-2013-11-07 10:26:57,836-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50625]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:26:57,836-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 10:26:57,836-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:26:57,836-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 10:26:57,836-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:26:57,836-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 10:26:57,836-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:26:57,836-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 10:26:57,836-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:26:57,837-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 10:26:57,837-0600 INFO ChannelManager Channel exception handled
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:26:57,837-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel config: null -> {}
-2013-11-07 10:26:57,837-0600 INFO AbstractCoasterChannel Channel config: null -> {}
-TCPChannel [type: server, contact: 127.0.0.1:50627] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:26:57,837-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50627] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-07 10:26:57,837-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-50627, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50627]] binding to NullChannel
-2013-11-07 10:26:57,837-0600 INFO MetaChannel MetaChannel [context: cps-50627, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50627]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50627]
-2013-11-07 10:26:57,837-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50627]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:26:57,837-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 10:26:57,837-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:26:57,837-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 10:26:57,837-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-07 10:26:57,843-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 10:26:57,844-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-07 10:26:57,844-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50625] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 10:26:57,844-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50625] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 10:26:57,844-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50625]
-2013-11-07 10:26:57,844-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50625]
-Channel is closed. Removing.
-2013-11-07 10:26:57,844-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 10:26:57,844-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {}
-2013-11-07 10:26:57,844-0600 INFO AbstractCoasterChannel Channel config: null -> {}
-TCPChannel [type: server, contact: 127.0.0.1:50627] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 10:26:57,844-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50627] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 10:26:57,844-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50627]
-2013-11-07 10:26:57,844-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50627]
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 10:26:57,851-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 10:26:57,852-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 10:26:57,852-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7526, JOBSTATUS) unregistering (send)
-2013-11-07 10:26:57,852-0600 INFO RequestHandler Handler(tag: 7526, JOBSTATUS) unregistering (send)
-2013-11-07 10:26:58,039-0600 DEBUG Cpu Completed
-crush.mcs.anl.gov:0 jobTerminated
-2013-11-07 10:26:58,039-0600 INFO Cpu crush.mcs.anl.gov:0 jobTerminated
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:26:58,039-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:26:58,039-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-Channel config: null -> {keepalive=null}
-2013-11-07 10:26:58,039-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-07 10:26:58,039-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-TCPChannel [type: server, contact: 127.0.0.1:50626] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:26:58,039-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50626] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 96, JOBSTATUS) unregistering (send)
-2013-11-07 10:26:58,039-0600 INFO RequestHandler Handler(tag: 96, JOBSTATUS) unregistering (send)
-null: Channel shut down
-2013-11-07 10:26:58,040-0600 INFO AbstractTCPChannel null: Channel shut down
-2013-11-07 10:26:58,039-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-MetaChannel [context: cps-50626, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50626]] binding to NullChannel
-2013-11-07 10:26:58,040-0600 INFO MetaChannel MetaChannel [context: cps-50626, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50626]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50626]
-2013-11-07 10:26:58,040-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50626]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:26:58,040-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 10:26:58,040-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:26:58,040-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 10:26:58,040-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:26:58,040-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 10:26:58,040-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:26:58,040-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 10:26:58,040-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:26:58,040-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 10:26:58,040-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-07 10:26:58,045-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 10:26:58,045-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-07 10:26:58,045-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50626] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 10:26:58,045-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50626] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 10:26:58,045-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50626]
-2013-11-07 10:26:58,045-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50626]
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 10:26:59,866-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 10:26:59,866-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 10:26:59,866-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 10:26:59,866-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-07 10:26:59,866-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7527, JOBSTATUS) unregistering (send)
-2013-11-07 10:26:59,866-0600 INFO RequestHandler Handler(tag: 7527, JOBSTATUS) unregistering (send)
-2013-11-07 10:26:59,866-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-No streams
-2013-11-07 10:27:02,538-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-07 10:27:09,593-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-07 10:27:12,546-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:27:22,555-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 10:27:31,016-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 10:27:32,562-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:27:42,570-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:27:46,068-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:27:52,578-0600 INFO Multiplexer No streams
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102557.934, now=131107-102758.778, channel=TCPChannel [type: server, contact: 127.0.0.1:50624]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:27:58,778-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102557.934, now=131107-102758.778, channel=TCPChannel [type: server, contact: 127.0.0.1:50624]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 10:27:58,779-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102557.934, now=131107-102758.778, channel=TCPChannel [type: server, contact: 127.0.0.1:50624]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:27:58,779-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102557.934, now=131107-102758.778, channel=TCPChannel [type: server, contact: 127.0.0.1:50624]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 10:27:58,779-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102557.934, now=131107-102758.778, channel=TCPChannel [type: server, contact: 127.0.0.1:50624]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:27:58,779-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102557.934, now=131107-102758.778, channel=TCPChannel [type: server, contact: 127.0.0.1:50624]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 10:27:58,779-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102557.934, now=131107-102758.778, channel=TCPChannel [type: server, contact: 127.0.0.1:50624]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:27:58,779-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102557.934, now=131107-102758.778, channel=TCPChannel [type: server, contact: 127.0.0.1:50624]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 10:27:58,779-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102557.934, now=131107-102758.778, channel=TCPChannel [type: server, contact: 127.0.0.1:50624]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:27:58,779-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102557.934, now=131107-102758.778, channel=TCPChannel [type: server, contact: 127.0.0.1:50624]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-07 10:27:58,779-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50624] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102557.934, now=131107-102758.778, channel=TCPChannel [type: server, contact: 127.0.0.1:50624]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:27:58,779-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50624] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102557.934, now=131107-102758.778, channel=TCPChannel [type: server, contact: 127.0.0.1:50624]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 10:27:58,779-0600 INFO ChannelManager Channel already shut down
-No streams
-2013-11-07 10:28:02,586-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:28:08,334-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:28:12,594-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:28:22,542-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:28:22,602-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:28:32,610-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:28:42,618-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:28:45,652-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:28:52,626-0600 INFO Multiplexer No streams
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102655.773, now=131107-102856.624, channel=TCPChannel [type: server, contact: 127.0.0.1:50625]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:28:56,624-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102655.773, now=131107-102856.624, channel=TCPChannel [type: server, contact: 127.0.0.1:50625]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 10:28:56,624-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102655.773, now=131107-102856.624, channel=TCPChannel [type: server, contact: 127.0.0.1:50625]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:28:56,624-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102655.773, now=131107-102856.624, channel=TCPChannel [type: server, contact: 127.0.0.1:50625]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 10:28:56,625-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102655.773, now=131107-102856.624, channel=TCPChannel [type: server, contact: 127.0.0.1:50625]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:28:56,625-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102655.773, now=131107-102856.624, channel=TCPChannel [type: server, contact: 127.0.0.1:50625]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 10:28:56,625-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102655.773, now=131107-102856.624, channel=TCPChannel [type: server, contact: 127.0.0.1:50625]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:28:56,625-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102655.773, now=131107-102856.624, channel=TCPChannel [type: server, contact: 127.0.0.1:50625]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 10:28:56,625-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102655.773, now=131107-102856.624, channel=TCPChannel [type: server, contact: 127.0.0.1:50625]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:28:56,625-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102655.773, now=131107-102856.624, channel=TCPChannel [type: server, contact: 127.0.0.1:50625]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-07 10:28:56,625-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50625] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102655.773, now=131107-102856.624, channel=TCPChannel [type: server, contact: 127.0.0.1:50625]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:28:56,625-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50625] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102655.773, now=131107-102856.624, channel=TCPChannel [type: server, contact: 127.0.0.1:50625]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 10:28:56,625-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102655.975, now=131107-102856.832, channel=TCPChannel [type: server, contact: 127.0.0.1:50626]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:28:56,832-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102655.975, now=131107-102856.832, channel=TCPChannel [type: server, contact: 127.0.0.1:50626]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 10:28:56,832-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102655.975, now=131107-102856.832, channel=TCPChannel [type: server, contact: 127.0.0.1:50626]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:28:56,832-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102655.975, now=131107-102856.832, channel=TCPChannel [type: server, contact: 127.0.0.1:50626]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 10:28:56,832-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102655.975, now=131107-102856.832, channel=TCPChannel [type: server, contact: 127.0.0.1:50626]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:28:56,832-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102655.975, now=131107-102856.832, channel=TCPChannel [type: server, contact: 127.0.0.1:50626]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 10:28:56,832-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102655.975, now=131107-102856.832, channel=TCPChannel [type: server, contact: 127.0.0.1:50626]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:28:56,832-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102655.975, now=131107-102856.832, channel=TCPChannel [type: server, contact: 127.0.0.1:50626]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 10:28:56,832-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102655.975, now=131107-102856.832, channel=TCPChannel [type: server, contact: 127.0.0.1:50626]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:28:56,832-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102655.975, now=131107-102856.832, channel=TCPChannel [type: server, contact: 127.0.0.1:50626]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-07 10:28:56,833-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50626] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102655.975, now=131107-102856.832, channel=TCPChannel [type: server, contact: 127.0.0.1:50626]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:28:56,833-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50626] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102655.975, now=131107-102856.832, channel=TCPChannel [type: server, contact: 127.0.0.1:50626]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 10:28:56,833-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102656.178, now=131107-102857.035, channel=TCPChannel [type: server, contact: 127.0.0.1:50627]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:28:57,035-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102656.178, now=131107-102857.035, channel=TCPChannel [type: server, contact: 127.0.0.1:50627]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 10:28:57,035-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102656.178, now=131107-102857.035, channel=TCPChannel [type: server, contact: 127.0.0.1:50627]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:28:57,035-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102656.178, now=131107-102857.035, channel=TCPChannel [type: server, contact: 127.0.0.1:50627]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {}
-2013-11-07 10:28:57,035-0600 INFO AbstractCoasterChannel Channel config: null -> {}
-TCPChannel [type: server, contact: 127.0.0.1:50627] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102656.178, now=131107-102857.035, channel=TCPChannel [type: server, contact: 127.0.0.1:50627]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:28:57,035-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50627] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-102656.178, now=131107-102857.035, channel=TCPChannel [type: server, contact: 127.0.0.1:50627]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 10:28:57,035-0600 INFO ChannelManager Channel already shut down
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-07 10:28:59,016-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-07 10:29:02,634-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:29:12,642-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:29:22,650-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 10:29:22,970-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 10:29:32,658-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:29:35,491-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:29:42,666-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:29:52,674-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:30:00,288-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:30:02,682-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:30:11,966-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:30:12,690-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:30:22,698-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:30:32,706-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:30:37,606-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:30:42,714-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 10:30:48,441-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 10:30:52,722-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:31:02,730-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:31:12,738-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 10:31:14,924-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 10:31:22,746-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-07 10:31:24,913-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-07 10:31:32,754-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:31:42,762-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:31:52,242-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:31:52,770-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:32:01,388-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:32:02,778-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:32:12,786-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:32:22,794-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 10:32:29,560-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 10:32:32,802-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:32:37,862-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:32:42,810-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:32:52,818-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:33:02,826-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:33:06,878-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:33:12,834-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:33:14,336-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:33:22,842-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:33:32,849-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:33:42,857-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:33:44,196-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:33:50,811-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:33:52,865-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:34:02,873-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:34:12,882-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 10:34:21,514-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 10:34:22,890-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:34:27,285-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:34:32,898-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:34:42,906-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:34:52,914-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:34:58,832-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:35:02,922-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 10:35:03,759-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 10:35:12,930-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:35:22,938-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:35:32,946-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 10:35:36,151-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:35:40,233-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:35:42,954-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:35:52,962-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:36:02,970-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:36:12,978-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:36:13,468-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 10:36:16,707-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 10:36:22,986-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:36:32,995-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:36:43,003-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-07 10:36:50,786-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-07 10:36:53,011-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:36:53,181-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:37:03,019-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:37:13,027-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:37:23,035-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:37:28,104-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:37:29,655-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:37:33,044-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:37:43,052-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:37:53,060-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:38:03,068-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:38:05,422-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:38:06,129-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:38:13,076-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:38:23,084-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:38:33,093-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:38:42,603-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 10:38:42,740-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 10:38:43,101-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:38:53,109-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:39:03,117-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:39:13,125-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:39:19,077-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:39:20,058-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:39:23,133-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:39:33,141-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:39:43,150-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:39:53,158-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 10:39:55,552-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-07 10:39:57,375-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-07 10:40:03,166-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:40:13,174-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:40:23,182-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:40:32,025-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:40:33,190-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:40:34,694-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:40:43,198-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:40:53,206-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:41:03,214-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:41:08,499-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:41:12,012-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:41:13,222-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:41:23,230-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:41:33,238-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:41:43,245-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:41:44,973-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 10:41:49,330-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 10:41:53,253-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:42:03,261-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:42:13,269-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:42:21,447-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:42:23,276-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:42:26,648-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:42:33,284-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:42:43,292-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:42:53,300-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:42:57,921-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:43:03,307-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-07 10:43:03,965-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-07 10:43:13,315-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:43:23,323-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:43:33,331-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:43:34,395-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:43:41,284-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:43:43,338-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:43:53,346-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:44:03,354-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:44:10,869-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:44:13,362-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:44:18,602-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:44:23,370-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:44:33,378-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:44:43,387-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 10:44:47,343-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 10:44:53,395-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 10:44:55,920-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 10:45:03,403-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:45:13,411-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:45:23,419-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:45:23,817-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:45:33,238-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:45:33,427-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:45:43,435-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:45:53,443-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:46:00,291-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:46:03,451-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:46:10,556-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:46:13,459-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:46:23,467-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:46:33,475-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 10:46:36,766-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 10:46:43,483-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 10:46:47,874-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 10:46:53,491-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:47:03,499-0600 INFO Multiplexer No streams
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 10:47:07,871-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50630] for addition
-2013-11-07 10:47:07,871-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50630] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:50630]
-2013-11-07 10:47:07,871-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:50630]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f71
-2013-11-07 10:47:07,892-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f71
-MetaChannel [context: cps-50630, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50630]
-2013-11-07 10:47:07,892-0600 INFO MetaChannel MetaChannel [context: cps-50630, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50630]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-07 10:47:07,892-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-07 10:47:07,933-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-07 10:47:07,933-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-07 10:47:08,013-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-07 10:47:08,033-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 98.0) submitting urn:job-0-1383759274876-1383759274877: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 10:47:08,034-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 98.0) submitting urn:job-0-1383759274876-1383759274877: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 10:47:08,034-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-07 10:47:08,034-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-07 10:47:08,040-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7528, JOBSTATUS) unregistering (send)
-2013-11-07 10:47:08,040-0600 INFO RequestHandler Handler(tag: 7528, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 10:47:08,072-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50631] for addition
-2013-11-07 10:47:08,072-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50631] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:50631]
-2013-11-07 10:47:08,072-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:50631]
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f70
-2013-11-07 10:47:08,094-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f70
-MetaChannel [context: cps-50631, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50631]
-2013-11-07 10:47:08,094-0600 INFO MetaChannel MetaChannel [context: cps-50631, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50631]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-07 10:47:08,094-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-07 10:47:08,155-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-07 10:47:08,235-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-07 10:47:08,284-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-crush.mcs.anl.gov:0 (quality: 33.0) submitting urn:job-1-1383759274879-1383759274880: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 10:47:08,284-0600 INFO Cpu crush.mcs.anl.gov:0 (quality: 33.0) submitting urn:job-1-1383759274879-1383759274880: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 10:47:08,290-0600 DEBUG Cpu Active workerid=crush.mcs.anl.gov:000000
-Handler(tag: 97, JOBSTATUS) unregistering (send)
-2013-11-07 10:47:08,290-0600 INFO RequestHandler Handler(tag: 97, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 10:47:08,296-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50632] for addition
-2013-11-07 10:47:08,296-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50632] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:50632]
-2013-11-07 10:47:08,296-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:50632]
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f6f
-2013-11-07 10:47:08,336-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f6f
-MetaChannel [context: cps-50632, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50632]
-2013-11-07 10:47:08,336-0600 INFO MetaChannel MetaChannel [context: cps-50632, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50632]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-07 10:47:08,336-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-07 10:47:08,377-0600 INFO RequestHandler Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-07 10:47:08,457-0600 INFO RequestHandler Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-07 10:47:10,066-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 10:47:10,066-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:47:10,066-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:47:10,066-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Channel config: null -> {keepalive=null}
-2013-11-07 10:47:10,066-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-07 10:47:10,066-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-Handler(tag: 7529, JOBSTATUS) unregistering (send)
-2013-11-07 10:47:10,066-0600 INFO RequestHandler Handler(tag: 7529, JOBSTATUS) unregistering (send)
-TCPChannel [type: server, contact: 127.0.0.1:50630] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:47:10,066-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50630] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-grind.mcs.anl.gov:0 (quality: 99.0) submitting urn:job-2-1383759274882-1383759274883: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 10:47:10,066-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 99.0) submitting urn:job-2-1383759274882-1383759274883: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-null: Channel shut down
-2013-11-07 10:47:10,066-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-50630, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50630]] binding to NullChannel
-2013-11-07 10:47:10,066-0600 INFO MetaChannel MetaChannel [context: cps-50630, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50630]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50630]
-2013-11-07 10:47:10,067-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50630]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:47:10,067-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 10:47:10,067-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:47:10,067-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 10:47:10,067-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:47:10,067-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 10:47:10,067-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:47:10,067-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 10:47:10,067-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:47:10,067-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 10:47:10,068-0600 INFO ChannelManager Channel exception handled
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:47:10,068-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel config: null -> {keepalive=null}
-2013-11-07 10:47:10,068-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50632] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:47:10,068-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50632] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-07 10:47:10,068-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-50632, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50632]] binding to NullChannel
-2013-11-07 10:47:10,068-0600 INFO MetaChannel MetaChannel [context: cps-50632, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50632]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50632]
-2013-11-07 10:47:10,068-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50632]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:47:10,068-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 10:47:10,068-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:47:10,068-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 10:47:10,068-0600 INFO ChannelManager Channel exception handled
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 10:47:10,071-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 10:47:10,072-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 10:47:10,072-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7530, JOBSTATUS) unregistering (send)
-2013-11-07 10:47:10,072-0600 INFO RequestHandler Handler(tag: 7530, JOBSTATUS) unregistering (send)
-Channel is closed. Removing.
-2013-11-07 10:47:10,082-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 10:47:10,082-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-07 10:47:10,082-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50630] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 10:47:10,082-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50630] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 10:47:10,082-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50630]
-2013-11-07 10:47:10,082-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50630]
-Channel is closed. Removing.
-2013-11-07 10:47:10,082-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 10:47:10,082-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-07 10:47:10,082-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50632] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 10:47:10,082-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50632] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 10:47:10,082-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50632]
-2013-11-07 10:47:10,082-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50632]
-2013-11-07 10:47:10,559-0600 DEBUG Cpu Completed
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:47:10,559-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-crush.mcs.anl.gov:0 jobTerminated
-2013-11-07 10:47:10,559-0600 INFO Cpu crush.mcs.anl.gov:0 jobTerminated
-Channel config: null -> {keepalive=null}
-2013-11-07 10:47:10,559-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-07 10:47:10,559-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-TCPChannel [type: server, contact: 127.0.0.1:50631] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:47:10,559-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50631] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 98, JOBSTATUS) unregistering (send)
-2013-11-07 10:47:10,559-0600 INFO RequestHandler Handler(tag: 98, JOBSTATUS) unregistering (send)
-2013-11-07 10:47:10,559-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-null: Channel shut down
-2013-11-07 10:47:10,560-0600 INFO AbstractTCPChannel null: Channel shut down
-2013-11-07 10:47:10,560-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-MetaChannel [context: cps-50631, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50631]] binding to NullChannel
-2013-11-07 10:47:10,560-0600 INFO MetaChannel MetaChannel [context: cps-50631, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50631]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50631]
-2013-11-07 10:47:10,560-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50631]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:47:10,560-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 10:47:10,560-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:47:10,560-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 10:47:10,560-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:47:10,560-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 10:47:10,560-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:47:10,560-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 10:47:10,560-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:47:10,560-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 10:47:10,560-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-07 10:47:10,564-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 10:47:10,564-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-07 10:47:10,564-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50631] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 10:47:10,564-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50631] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 10:47:10,564-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50631]
-2013-11-07 10:47:10,564-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50631]
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 10:47:12,085-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 10:47:12,086-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 10:47:12,086-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 10:47:12,086-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-07 10:47:12,086-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7531, JOBSTATUS) unregistering (send)
-2013-11-07 10:47:12,086-0600 INFO RequestHandler Handler(tag: 7531, JOBSTATUS) unregistering (send)
-2013-11-07 10:47:12,086-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-07 10:47:13,239-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-07 10:47:13,512-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:47:23,521-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:47:25,192-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:47:33,529-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:47:43,537-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:47:49,713-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:47:53,545-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:48:02,510-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:48:03,554-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:48:13,562-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:48:23,570-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 10:48:26,187-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 10:48:33,578-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 10:48:39,828-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 10:48:43,587-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:48:53,595-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 10:49:02,662-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 10:49:03,603-0600 INFO Multiplexer No streams
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-104708.013, now=131107-104908.871, channel=TCPChannel [type: server, contact: 127.0.0.1:50630]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:49:08,871-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-104708.013, now=131107-104908.871, channel=TCPChannel [type: server, contact: 127.0.0.1:50630]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 10:49:08,871-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-104708.013, now=131107-104908.871, channel=TCPChannel [type: server, contact: 127.0.0.1:50630]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:49:08,871-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-104708.013, now=131107-104908.871, channel=TCPChannel [type: server, contact: 127.0.0.1:50630]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 10:49:08,871-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-104708.013, now=131107-104908.871, channel=TCPChannel [type: server, contact: 127.0.0.1:50630]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:49:08,871-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-104708.013, now=131107-104908.871, channel=TCPChannel [type: server, contact: 127.0.0.1:50630]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 10:49:08,871-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-104708.013, now=131107-104908.871, channel=TCPChannel [type: server, contact: 127.0.0.1:50630]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:49:08,871-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-104708.013, now=131107-104908.871, channel=TCPChannel [type: server, contact: 127.0.0.1:50630]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 10:49:08,872-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-104708.013, now=131107-104908.871, channel=TCPChannel [type: server, contact: 127.0.0.1:50630]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:49:08,872-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-104708.013, now=131107-104908.871, channel=TCPChannel [type: server, contact: 127.0.0.1:50630]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-07 10:49:08,872-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50630] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-104708.013, now=131107-104908.871, channel=TCPChannel [type: server, contact: 127.0.0.1:50630]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:49:08,872-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50630] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-104708.013, now=131107-104908.871, channel=TCPChannel [type: server, contact: 127.0.0.1:50630]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 10:49:08,872-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-104708.235, now=131107-104909.072, channel=TCPChannel [type: server, contact: 127.0.0.1:50631]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:49:09,072-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-104708.235, now=131107-104909.072, channel=TCPChannel [type: server, contact: 127.0.0.1:50631]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 10:49:09,072-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-104708.235, now=131107-104909.072, channel=TCPChannel [type: server, contact: 127.0.0.1:50631]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:49:09,072-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-104708.235, now=131107-104909.072, channel=TCPChannel [type: server, contact: 127.0.0.1:50631]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 10:49:09,072-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-104708.235, now=131107-104909.072, channel=TCPChannel [type: server, contact: 127.0.0.1:50631]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:49:09,072-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-104708.235, now=131107-104909.072, channel=TCPChannel [type: server, contact: 127.0.0.1:50631]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 10:49:09,072-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-104708.235, now=131107-104909.072, channel=TCPChannel [type: server, contact: 127.0.0.1:50631]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:49:09,072-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-104708.235, now=131107-104909.072, channel=TCPChannel [type: server, contact: 127.0.0.1:50631]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 10:49:09,072-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-104708.235, now=131107-104909.072, channel=TCPChannel [type: server, contact: 127.0.0.1:50631]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:49:09,072-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-104708.235, now=131107-104909.072, channel=TCPChannel [type: server, contact: 127.0.0.1:50631]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-07 10:49:09,072-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50631] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-104708.235, now=131107-104909.072, channel=TCPChannel [type: server, contact: 127.0.0.1:50631]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:49:09,072-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50631] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-104708.235, now=131107-104909.072, channel=TCPChannel [type: server, contact: 127.0.0.1:50631]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 10:49:09,072-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-104708.457, now=131107-104909.296, channel=TCPChannel [type: server, contact: 127.0.0.1:50632]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:49:09,296-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-104708.457, now=131107-104909.296, channel=TCPChannel [type: server, contact: 127.0.0.1:50632]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 10:49:09,296-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-104708.457, now=131107-104909.296, channel=TCPChannel [type: server, contact: 127.0.0.1:50632]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:49:09,296-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-104708.457, now=131107-104909.296, channel=TCPChannel [type: server, contact: 127.0.0.1:50632]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-07 10:49:09,296-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50632] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-104708.457, now=131107-104909.296, channel=TCPChannel [type: server, contact: 127.0.0.1:50632]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:49:09,296-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50632] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-104708.457, now=131107-104909.296, channel=TCPChannel [type: server, contact: 127.0.0.1:50632]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 10:49:09,296-0600 INFO ChannelManager Channel already shut down
-No streams
-2013-11-07 10:49:13,611-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:49:17,146-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:49:23,619-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:49:33,627-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:49:39,135-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:49:43,635-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:49:53,643-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-07 10:49:54,464-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-07 10:50:03,650-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:50:13,658-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:50:15,609-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:50:23,666-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:50:31,782-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:50:33,674-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:50:43,682-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:50:52,084-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:50:53,690-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:51:03,698-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-07 10:51:09,100-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-07 10:51:13,706-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:51:23,714-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:51:28,558-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:51:33,722-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:51:43,729-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:51:46,418-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:51:53,737-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:52:03,745-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:52:05,032-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:52:13,753-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:52:23,736-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:52:23,761-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:52:33,769-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:52:41,506-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:52:43,777-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:52:53,785-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 10:53:01,055-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 10:53:03,793-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:53:13,801-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 10:53:17,981-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 10:53:23,809-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:53:33,817-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:53:38,372-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:53:43,824-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:53:53,832-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:53:54,454-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:54:03,840-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:54:13,848-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 10:54:15,690-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 10:54:23,856-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:54:30,928-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:54:33,864-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:54:43,873-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:54:53,008-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:54:53,881-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:55:03,889-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:55:07,402-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:55:13,897-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:55:23,905-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:55:30,327-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:55:33,913-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 3ms
-2013-11-07 10:55:43,878-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 3ms
-No streams
-2013-11-07 10:55:43,921-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:55:53,929-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:56:03,938-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 10:56:07,645-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 10:56:13,946-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:56:20,350-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:56:23,954-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:56:33,962-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:56:43,970-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:56:44,963-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 10:56:50,844-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50634] for addition
-2013-11-07 10:56:50,844-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50634] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:50634]
-2013-11-07 10:56:50,844-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:50634]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f6e
-2013-11-07 10:56:50,870-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f6e
-MetaChannel [context: cps-50634, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50634]
-2013-11-07 10:56:50,870-0600 INFO MetaChannel MetaChannel [context: cps-50634, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50634]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-07 10:56:50,870-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-07 10:56:50,911-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-07 10:56:50,911-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-07 10:56:50,991-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-07 10:56:50,993-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 100.0) submitting urn:job-0-1383759274885-1383759274886: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 10:56:50,993-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 100.0) submitting urn:job-0-1383759274885-1383759274886: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 10:56:50,994-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-07 10:56:50,994-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-07 10:56:50,999-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7532, JOBSTATUS) unregistering (send)
-2013-11-07 10:56:50,999-0600 INFO RequestHandler Handler(tag: 7532, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 10:56:51,052-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50635] for addition
-2013-11-07 10:56:51,052-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50635] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:50635]
-2013-11-07 10:56:51,053-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:50635]
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f6d
-2013-11-07 10:56:51,092-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f6d
-MetaChannel [context: cps-50635, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50635]
-2013-11-07 10:56:51,092-0600 INFO MetaChannel MetaChannel [context: cps-50635, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50635]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-07 10:56:51,092-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-07 10:56:51,113-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-07 10:56:51,193-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-07 10:56:51,194-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-crush.mcs.anl.gov:0 (quality: 34.0) submitting urn:job-1-1383759274888-1383759274889: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 10:56:51,194-0600 INFO Cpu crush.mcs.anl.gov:0 (quality: 34.0) submitting urn:job-1-1383759274888-1383759274889: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 10:56:51,209-0600 DEBUG Cpu Active workerid=crush.mcs.anl.gov:000000
-Handler(tag: 99, JOBSTATUS) unregistering (send)
-2013-11-07 10:56:51,209-0600 INFO RequestHandler Handler(tag: 99, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 10:56:51,251-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50636] for addition
-2013-11-07 10:56:51,251-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50636] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:50636]
-2013-11-07 10:56:51,251-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:50636]
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f6c
-2013-11-07 10:56:51,274-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f6c
-MetaChannel [context: cps-50636, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50636]
-2013-11-07 10:56:51,274-0600 INFO MetaChannel MetaChannel [context: cps-50636, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50636]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-07 10:56:51,274-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-07 10:56:51,334-0600 INFO RequestHandler Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-07 10:56:51,415-0600 INFO RequestHandler Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-07 10:56:53,212-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 10:56:53,212-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:56:53,212-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:56:53,212-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Channel config: null -> {keepalive=null}
-2013-11-07 10:56:53,212-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-07 10:56:53,212-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-Handler(tag: 7533, JOBSTATUS) unregistering (send)
-2013-11-07 10:56:53,212-0600 INFO RequestHandler Handler(tag: 7533, JOBSTATUS) unregistering (send)
-TCPChannel [type: server, contact: 127.0.0.1:50634] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:56:53,212-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50634] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-grind.mcs.anl.gov:0 (quality: 101.0) submitting urn:job-2-1383759274891-1383759274892: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 10:56:53,213-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 101.0) submitting urn:job-2-1383759274891-1383759274892: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-null: Channel shut down
-2013-11-07 10:56:53,213-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-50634, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50634]] binding to NullChannel
-2013-11-07 10:56:53,213-0600 INFO MetaChannel MetaChannel [context: cps-50634, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50634]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50634]
-2013-11-07 10:56:53,213-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50634]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:56:53,213-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 10:56:53,213-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:56:53,213-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 10:56:53,213-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:56:53,213-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 10:56:53,214-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:56:53,214-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 10:56:53,214-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:56:53,214-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 10:56:53,214-0600 INFO ChannelManager Channel exception handled
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:56:53,214-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel config: null -> {keepalive=null}
-2013-11-07 10:56:53,214-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50636] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:56:53,214-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50636] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-07 10:56:53,214-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-50636, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50636]] binding to NullChannel
-2013-11-07 10:56:53,214-0600 INFO MetaChannel MetaChannel [context: cps-50636, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50636]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50636]
-2013-11-07 10:56:53,214-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50636]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:56:53,214-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 10:56:53,215-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:56:53,215-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 10:56:53,215-0600 INFO ChannelManager Channel exception handled
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 10:56:53,218-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 10:56:53,219-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 10:56:53,219-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7534, JOBSTATUS) unregistering (send)
-2013-11-07 10:56:53,219-0600 INFO RequestHandler Handler(tag: 7534, JOBSTATUS) unregistering (send)
-Channel is closed. Removing.
-2013-11-07 10:56:53,220-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 10:56:53,221-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-07 10:56:53,221-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50634] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 10:56:53,221-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50634] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 10:56:53,221-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50634]
-2013-11-07 10:56:53,221-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50634]
-Channel is closed. Removing.
-2013-11-07 10:56:53,221-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 10:56:53,221-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-07 10:56:53,221-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50636] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 10:56:53,221-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50636] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 10:56:53,221-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50636]
-2013-11-07 10:56:53,221-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50636]
-2013-11-07 10:56:53,570-0600 DEBUG Cpu Completed
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:56:53,570-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-crush.mcs.anl.gov:0 jobTerminated
-2013-11-07 10:56:53,570-0600 INFO Cpu crush.mcs.anl.gov:0 jobTerminated
-Channel config: null -> {keepalive=null}
-2013-11-07 10:56:53,570-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-07 10:56:53,570-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-TCPChannel [type: server, contact: 127.0.0.1:50635] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:56:53,570-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50635] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 100, JOBSTATUS) unregistering (send)
-2013-11-07 10:56:53,570-0600 INFO RequestHandler Handler(tag: 100, JOBSTATUS) unregistering (send)
-2013-11-07 10:56:53,570-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-null: Channel shut down
-2013-11-07 10:56:53,570-0600 INFO AbstractTCPChannel null: Channel shut down
-2013-11-07 10:56:53,570-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-MetaChannel [context: cps-50635, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50635]] binding to NullChannel
-2013-11-07 10:56:53,570-0600 INFO MetaChannel MetaChannel [context: cps-50635, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50635]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50635]
-2013-11-07 10:56:53,570-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50635]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:56:53,571-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 10:56:53,571-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:56:53,571-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 10:56:53,571-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:56:53,571-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 10:56:53,571-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:56:53,571-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 10:56:53,571-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:56:53,571-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 10:56:53,571-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-07 10:56:53,582-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 10:56:53,582-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-07 10:56:53,582-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50635] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 10:56:53,582-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50635] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 10:56:53,582-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50635]
-2013-11-07 10:56:53,582-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50635]
-No streams
-2013-11-07 10:56:53,984-0600 INFO Multiplexer No streams
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 10:56:55,233-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 10:56:55,233-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 10:56:55,233-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 10:56:55,233-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-07 10:56:55,233-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7535, JOBSTATUS) unregistering (send)
-2013-11-07 10:56:55,233-0600 INFO RequestHandler Handler(tag: 7535, JOBSTATUS) unregistering (send)
-2013-11-07 10:56:55,233-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:56:56,824-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:57:03,992-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:57:14,000-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:57:22,281-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:57:24,008-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:57:33,298-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:57:34,016-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:57:44,024-0600 INFO Multiplexer No streams
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 10:57:49,070-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50643] for addition
-2013-11-07 10:57:49,070-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50643] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:50643]
-2013-11-07 10:57:49,070-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:50643]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f6b
-2013-11-07 10:57:49,099-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f6b
-MetaChannel [context: cps-50643, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50643]
-2013-11-07 10:57:49,099-0600 INFO MetaChannel MetaChannel [context: cps-50643, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50643]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-07 10:57:49,099-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-07 10:57:49,139-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-07 10:57:49,139-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-07 10:57:49,220-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-07 10:57:49,245-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 102.0) submitting urn:job-0-1383759274894-1383759274895: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 10:57:49,245-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 102.0) submitting urn:job-0-1383759274894-1383759274895: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 10:57:49,246-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-07 10:57:49,246-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-07 10:57:49,251-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7536, JOBSTATUS) unregistering (send)
-2013-11-07 10:57:49,251-0600 INFO RequestHandler Handler(tag: 7536, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 10:57:49,280-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50644] for addition
-2013-11-07 10:57:49,280-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50644] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:50644]
-2013-11-07 10:57:49,280-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:50644]
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f6a
-2013-11-07 10:57:49,301-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f6a
-MetaChannel [context: cps-50644, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50644]
-2013-11-07 10:57:49,301-0600 INFO MetaChannel MetaChannel [context: cps-50644, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50644]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-07 10:57:49,301-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-07 10:57:49,341-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-07 10:57:49,422-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-07 10:57:49,446-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-crush.mcs.anl.gov:0 (quality: 35.0) submitting urn:job-1-1383759274897-1383759274898: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 10:57:49,447-0600 INFO Cpu crush.mcs.anl.gov:0 (quality: 35.0) submitting urn:job-1-1383759274897-1383759274898: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 10:57:49,453-0600 DEBUG Cpu Active workerid=crush.mcs.anl.gov:000000
-Handler(tag: 101, JOBSTATUS) unregistering (send)
-2013-11-07 10:57:49,453-0600 INFO RequestHandler Handler(tag: 101, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 10:57:49,480-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50645] for addition
-2013-11-07 10:57:49,480-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50645] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:50645]
-2013-11-07 10:57:49,480-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:50645]
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f69
-2013-11-07 10:57:49,502-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f69
-MetaChannel [context: cps-50645, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50645]
-2013-11-07 10:57:49,503-0600 INFO MetaChannel MetaChannel [context: cps-50645, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50645]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-07 10:57:49,503-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-07 10:57:49,543-0600 INFO RequestHandler Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-07 10:57:49,624-0600 INFO RequestHandler Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-07 10:57:51,281-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 10:57:51,281-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:57:51,281-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:57:51,281-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Channel config: null -> {keepalive=null}
-2013-11-07 10:57:51,281-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-07 10:57:51,281-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-Handler(tag: 7537, JOBSTATUS) unregistering (send)
-2013-11-07 10:57:51,281-0600 INFO RequestHandler Handler(tag: 7537, JOBSTATUS) unregistering (send)
-TCPChannel [type: server, contact: 127.0.0.1:50643] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:57:51,281-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50643] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-grind.mcs.anl.gov:0 (quality: 103.0) submitting urn:job-2-1383759274900-1383759274901: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 10:57:51,282-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 103.0) submitting urn:job-2-1383759274900-1383759274901: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-null: Channel shut down
-2013-11-07 10:57:51,282-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-50643, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50643]] binding to NullChannel
-2013-11-07 10:57:51,282-0600 INFO MetaChannel MetaChannel [context: cps-50643, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50643]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50643]
-2013-11-07 10:57:51,282-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50643]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:57:51,282-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 10:57:51,283-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:57:51,283-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 10:57:51,283-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:57:51,283-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 10:57:51,283-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:57:51,283-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 10:57:51,283-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:57:51,283-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 10:57:51,283-0600 INFO ChannelManager Channel exception handled
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:57:51,283-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel config: null -> {keepalive=null}
-2013-11-07 10:57:51,283-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50645] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:57:51,283-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50645] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-07 10:57:51,283-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-50645, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50645]] binding to NullChannel
-2013-11-07 10:57:51,284-0600 INFO MetaChannel MetaChannel [context: cps-50645, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50645]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50645]
-2013-11-07 10:57:51,284-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50645]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:57:51,284-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 10:57:51,284-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:57:51,284-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 10:57:51,284-0600 INFO ChannelManager Channel exception handled
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 10:57:51,287-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 10:57:51,288-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 10:57:51,288-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7538, JOBSTATUS) unregistering (send)
-2013-11-07 10:57:51,288-0600 INFO RequestHandler Handler(tag: 7538, JOBSTATUS) unregistering (send)
-Channel is closed. Removing.
-2013-11-07 10:57:51,289-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 10:57:51,289-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-07 10:57:51,289-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50645] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 10:57:51,289-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50645] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 10:57:51,289-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50645]
-2013-11-07 10:57:51,289-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50645]
-Channel is closed. Removing.
-2013-11-07 10:57:51,289-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 10:57:51,289-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-07 10:57:51,289-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50643] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 10:57:51,289-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50643] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 10:57:51,289-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50643]
-2013-11-07 10:57:51,290-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50643]
-2013-11-07 10:57:51,471-0600 DEBUG Cpu Completed
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:57:51,471-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-crush.mcs.anl.gov:0 jobTerminated
-2013-11-07 10:57:51,471-0600 INFO Cpu crush.mcs.anl.gov:0 jobTerminated
-Channel config: null -> {keepalive=null}
-2013-11-07 10:57:51,471-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-07 10:57:51,471-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-TCPChannel [type: server, contact: 127.0.0.1:50644] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:57:51,472-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50644] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:57:51,472-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-null: Channel shut down
-2013-11-07 10:57:51,472-0600 INFO AbstractTCPChannel null: Channel shut down
-Handler(tag: 102, JOBSTATUS) unregistering (send)
-2013-11-07 10:57:51,472-0600 INFO RequestHandler Handler(tag: 102, JOBSTATUS) unregistering (send)
-MetaChannel [context: cps-50644, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50644]] binding to NullChannel
-2013-11-07 10:57:51,472-0600 INFO MetaChannel MetaChannel [context: cps-50644, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50644]] binding to NullChannel
-2013-11-07 10:57:51,472-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50644]
-2013-11-07 10:57:51,472-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50644]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:57:51,472-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 10:57:51,472-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:57:51,472-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 10:57:51,472-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:57:51,472-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 10:57:51,472-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:57:51,473-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 10:57:51,473-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:57:51,473-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 10:57:51,473-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-07 10:57:51,490-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 10:57:51,490-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-07 10:57:51,490-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50644] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 10:57:51,490-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50644] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 10:57:51,490-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50644]
-2013-11-07 10:57:51,490-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50644]
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 10:57:53,512-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 10:57:53,512-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 10:57:53,512-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 10:57:53,512-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-07 10:57:53,512-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7539, JOBSTATUS) unregistering (send)
-2013-11-07 10:57:53,512-0600 INFO RequestHandler Handler(tag: 7539, JOBSTATUS) unregistering (send)
-2013-11-07 10:57:53,512-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-No streams
-2013-11-07 10:57:54,038-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:57:59,599-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:58:04,045-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:58:09,772-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:58:14,053-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:58:24,060-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:58:34,068-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 10:58:36,918-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 10:58:38,006-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50646] for addition
-2013-11-07 10:58:38,007-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50646] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:50646]
-2013-11-07 10:58:38,007-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:50646]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f68
-2013-11-07 10:58:38,040-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f68
-MetaChannel [context: cps-50646, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50646]
-2013-11-07 10:58:38,040-0600 INFO MetaChannel MetaChannel [context: cps-50646, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50646]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-07 10:58:38,040-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-07 10:58:38,080-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-07 10:58:38,081-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-07 10:58:38,161-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-07 10:58:38,164-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 104.0) submitting urn:job-0-1383759274903-1383759274904: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 10:58:38,164-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 104.0) submitting urn:job-0-1383759274903-1383759274904: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 10:58:38,164-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-07 10:58:38,164-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-07 10:58:38,169-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7540, JOBSTATUS) unregistering (send)
-2013-11-07 10:58:38,169-0600 INFO RequestHandler Handler(tag: 7540, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 10:58:38,220-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50647] for addition
-2013-11-07 10:58:38,220-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50647] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:50647]
-2013-11-07 10:58:38,220-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:50647]
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f67
-2013-11-07 10:58:38,247-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f67
-MetaChannel [context: cps-50647, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50647]
-2013-11-07 10:58:38,247-0600 INFO MetaChannel MetaChannel [context: cps-50647, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50647]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-07 10:58:38,247-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-07 10:58:38,287-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-07 10:58:38,368-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-07 10:58:38,415-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-crush.mcs.anl.gov:0 (quality: 36.0) submitting urn:job-1-1383759274906-1383759274907: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 10:58:38,415-0600 INFO Cpu crush.mcs.anl.gov:0 (quality: 36.0) submitting urn:job-1-1383759274906-1383759274907: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 10:58:38,429-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50648] for addition
-2013-11-07 10:58:38,429-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50648] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:50648]
-2013-11-07 10:58:38,429-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:50648]
-2013-11-07 10:58:38,435-0600 DEBUG Cpu Active workerid=crush.mcs.anl.gov:000000
-Handler(tag: 103, JOBSTATUS) unregistering (send)
-2013-11-07 10:58:38,435-0600 INFO RequestHandler Handler(tag: 103, JOBSTATUS) unregistering (send)
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f66
-2013-11-07 10:58:38,469-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f66
-MetaChannel [context: cps-50648, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50648]
-2013-11-07 10:58:38,469-0600 INFO MetaChannel MetaChannel [context: cps-50648, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50648]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-07 10:58:38,469-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-07 10:58:38,489-0600 INFO RequestHandler Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-07 10:58:38,569-0600 INFO RequestHandler Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-07 10:58:40,234-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 10:58:40,234-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 10:58:40,234-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:58:40,234-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel config: null -> {keepalive=null}
-2013-11-07 10:58:40,234-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50646] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:58:40,234-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50646] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:58:40,234-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-Handler(tag: 7541, JOBSTATUS) unregistering (send)
-2013-11-07 10:58:40,234-0600 INFO RequestHandler Handler(tag: 7541, JOBSTATUS) unregistering (send)
-null: Channel shut down
-2013-11-07 10:58:40,234-0600 INFO AbstractTCPChannel null: Channel shut down
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 10:58:40,234-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-grind.mcs.anl.gov:0 (quality: 105.0) submitting urn:job-2-1383759274909-1383759274910: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 10:58:40,234-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 105.0) submitting urn:job-2-1383759274909-1383759274910: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-MetaChannel [context: cps-50646, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50646]] binding to NullChannel
-2013-11-07 10:58:40,234-0600 INFO MetaChannel MetaChannel [context: cps-50646, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50646]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50646]
-2013-11-07 10:58:40,235-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50646]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:58:40,235-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 10:58:40,235-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:58:40,235-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 10:58:40,235-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:58:40,235-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 10:58:40,236-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:58:40,236-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 10:58:40,236-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:58:40,236-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 10:58:40,236-0600 INFO ChannelManager Channel exception handled
-Channel config: null -> {keepalive=null}
-2013-11-07 10:58:40,236-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:58:40,236-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-TCPChannel [type: server, contact: 127.0.0.1:50646] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 10:58:40,236-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50646] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 10:58:40,236-0600 INFO ChannelManager Channel already shut down
-Channel config: null -> {keepalive=null}
-2013-11-07 10:58:40,236-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50646]
-2013-11-07 10:58:40,236-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50646]
-TCPChannel [type: server, contact: 127.0.0.1:50648] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:58:40,236-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50648] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-07 10:58:40,237-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-50648, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50648]] binding to NullChannel
-2013-11-07 10:58:40,237-0600 INFO MetaChannel MetaChannel [context: cps-50648, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50648]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50648]
-2013-11-07 10:58:40,237-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50648]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:58:40,237-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 10:58:40,237-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:58:40,237-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 10:58:40,237-0600 INFO ChannelManager Channel exception handled
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 10:58:40,240-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 10:58:40,241-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 10:58:40,241-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7542, JOBSTATUS) unregistering (send)
-2013-11-07 10:58:40,241-0600 INFO RequestHandler Handler(tag: 7542, JOBSTATUS) unregistering (send)
-Channel is closed. Removing.
-2013-11-07 10:58:40,257-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 10:58:40,257-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-07 10:58:40,257-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50648] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 10:58:40,257-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50648] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 10:58:40,257-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50648]
-2013-11-07 10:58:40,257-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50648]
-2013-11-07 10:58:40,605-0600 DEBUG Cpu Completed
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:58:40,606-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-crush.mcs.anl.gov:0 jobTerminated
-2013-11-07 10:58:40,606-0600 INFO Cpu crush.mcs.anl.gov:0 jobTerminated
-Channel config: null -> {keepalive=null}
-2013-11-07 10:58:40,606-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-07 10:58:40,606-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-TCPChannel [type: server, contact: 127.0.0.1:50647] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:58:40,606-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50647] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:58:40,606-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-null: Channel shut down
-2013-11-07 10:58:40,606-0600 INFO AbstractTCPChannel null: Channel shut down
-Handler(tag: 104, JOBSTATUS) unregistering (send)
-2013-11-07 10:58:40,606-0600 INFO RequestHandler Handler(tag: 104, JOBSTATUS) unregistering (send)
-MetaChannel [context: cps-50647, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50647]] binding to NullChannel
-2013-11-07 10:58:40,606-0600 INFO MetaChannel MetaChannel [context: cps-50647, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50647]] binding to NullChannel
-2013-11-07 10:58:40,606-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50647]
-2013-11-07 10:58:40,606-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50647]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:58:40,606-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 10:58:40,606-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:58:40,606-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 10:58:40,607-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:58:40,607-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 10:58:40,607-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:58:40,607-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 10:58:40,607-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:58:40,607-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 10:58:40,607-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-07 10:58:40,618-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 10:58:40,618-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-07 10:58:40,618-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50647] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 10:58:40,618-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50647] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 10:58:40,618-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50647]
-2013-11-07 10:58:40,618-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50647]
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 10:58:42,254-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 10:58:42,254-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 10:58:42,254-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 10:58:42,254-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-07 10:58:42,254-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7543, JOBSTATUS) unregistering (send)
-2013-11-07 10:58:42,254-0600 INFO RequestHandler Handler(tag: 7543, JOBSTATUS) unregistering (send)
-2013-11-07 10:58:42,254-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-No streams
-2013-11-07 10:58:44,088-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:58:46,247-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105650.991, now=131107-105851.844, channel=TCPChannel [type: server, contact: 127.0.0.1:50634]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:58:51,844-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105650.991, now=131107-105851.844, channel=TCPChannel [type: server, contact: 127.0.0.1:50634]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 10:58:51,844-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105650.991, now=131107-105851.844, channel=TCPChannel [type: server, contact: 127.0.0.1:50634]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:58:51,844-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105650.991, now=131107-105851.844, channel=TCPChannel [type: server, contact: 127.0.0.1:50634]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 10:58:51,844-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105650.991, now=131107-105851.844, channel=TCPChannel [type: server, contact: 127.0.0.1:50634]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:58:51,844-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105650.991, now=131107-105851.844, channel=TCPChannel [type: server, contact: 127.0.0.1:50634]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 10:58:51,845-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105650.991, now=131107-105851.844, channel=TCPChannel [type: server, contact: 127.0.0.1:50634]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:58:51,845-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105650.991, now=131107-105851.844, channel=TCPChannel [type: server, contact: 127.0.0.1:50634]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 10:58:51,845-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105650.991, now=131107-105851.844, channel=TCPChannel [type: server, contact: 127.0.0.1:50634]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:58:51,845-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105650.991, now=131107-105851.844, channel=TCPChannel [type: server, contact: 127.0.0.1:50634]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-07 10:58:51,845-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50634] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105650.991, now=131107-105851.844, channel=TCPChannel [type: server, contact: 127.0.0.1:50634]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:58:51,845-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50634] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105650.991, now=131107-105851.844, channel=TCPChannel [type: server, contact: 127.0.0.1:50634]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 10:58:51,845-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105651.193, now=131107-105852.052, channel=TCPChannel [type: server, contact: 127.0.0.1:50635]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:58:52,052-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105651.193, now=131107-105852.052, channel=TCPChannel [type: server, contact: 127.0.0.1:50635]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 10:58:52,052-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105651.193, now=131107-105852.052, channel=TCPChannel [type: server, contact: 127.0.0.1:50635]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:58:52,052-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105651.193, now=131107-105852.052, channel=TCPChannel [type: server, contact: 127.0.0.1:50635]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 10:58:52,052-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105651.193, now=131107-105852.052, channel=TCPChannel [type: server, contact: 127.0.0.1:50635]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:58:52,052-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105651.193, now=131107-105852.052, channel=TCPChannel [type: server, contact: 127.0.0.1:50635]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 10:58:52,052-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105651.193, now=131107-105852.052, channel=TCPChannel [type: server, contact: 127.0.0.1:50635]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:58:52,052-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105651.193, now=131107-105852.052, channel=TCPChannel [type: server, contact: 127.0.0.1:50635]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 10:58:52,052-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105651.193, now=131107-105852.052, channel=TCPChannel [type: server, contact: 127.0.0.1:50635]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:58:52,052-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105651.193, now=131107-105852.052, channel=TCPChannel [type: server, contact: 127.0.0.1:50635]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-07 10:58:52,053-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50635] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105651.193, now=131107-105852.052, channel=TCPChannel [type: server, contact: 127.0.0.1:50635]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:58:52,053-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50635] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105651.193, now=131107-105852.052, channel=TCPChannel [type: server, contact: 127.0.0.1:50635]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 10:58:52,053-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105651.415, now=131107-105852.251, channel=TCPChannel [type: server, contact: 127.0.0.1:50636]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:58:52,251-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105651.415, now=131107-105852.251, channel=TCPChannel [type: server, contact: 127.0.0.1:50636]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 10:58:52,251-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105651.415, now=131107-105852.251, channel=TCPChannel [type: server, contact: 127.0.0.1:50636]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:58:52,251-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105651.415, now=131107-105852.251, channel=TCPChannel [type: server, contact: 127.0.0.1:50636]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-07 10:58:52,251-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50636] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105651.415, now=131107-105852.251, channel=TCPChannel [type: server, contact: 127.0.0.1:50636]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:58:52,251-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50636] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105651.415, now=131107-105852.251, channel=TCPChannel [type: server, contact: 127.0.0.1:50636]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 10:58:52,251-0600 INFO ChannelManager Channel already shut down
-No streams
-2013-11-07 10:58:54,096-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:59:04,105-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:59:14,113-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 10:59:14,236-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 10:59:22,721-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 10:59:24,121-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 10:59:34,129-0600 INFO Multiplexer No streams
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 10:59:42,038-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50649] for addition
-2013-11-07 10:59:42,038-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50649] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:50649]
-2013-11-07 10:59:42,038-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:50649]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f65
-2013-11-07 10:59:42,072-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f65
-MetaChannel [context: cps-50649, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50649]
-2013-11-07 10:59:42,072-0600 INFO MetaChannel MetaChannel [context: cps-50649, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50649]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-07 10:59:42,072-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-07 10:59:42,113-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-07 10:59:42,113-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-07 10:59:42,193-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-07 10:59:42,223-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 106.0) submitting urn:job-0-1383759274912-1383759274913: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 10:59:42,223-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 106.0) submitting urn:job-0-1383759274912-1383759274913: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 10:59:42,223-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-07 10:59:42,223-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-07 10:59:42,228-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7544, JOBSTATUS) unregistering (send)
-2013-11-07 10:59:42,229-0600 INFO RequestHandler Handler(tag: 7544, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 10:59:42,252-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50650] for addition
-2013-11-07 10:59:42,252-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50650] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:50650]
-2013-11-07 10:59:42,252-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:50650]
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f64
-2013-11-07 10:59:42,274-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f64
-MetaChannel [context: cps-50650, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50650]
-2013-11-07 10:59:42,274-0600 INFO MetaChannel MetaChannel [context: cps-50650, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50650]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-07 10:59:42,274-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-07 10:59:42,335-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-07 10:59:42,415-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-07 10:59:42,424-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-crush.mcs.anl.gov:0 (quality: 37.0) submitting urn:job-1-1383759274915-1383759274916: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 10:59:42,424-0600 INFO Cpu crush.mcs.anl.gov:0 (quality: 37.0) submitting urn:job-1-1383759274915-1383759274916: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 10:59:42,429-0600 DEBUG Cpu Active workerid=crush.mcs.anl.gov:000000
-Handler(tag: 105, JOBSTATUS) unregistering (send)
-2013-11-07 10:59:42,429-0600 INFO RequestHandler Handler(tag: 105, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 10:59:42,476-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50651] for addition
-2013-11-07 10:59:42,476-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50651] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:50651]
-2013-11-07 10:59:42,476-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:50651]
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f63
-2013-11-07 10:59:42,516-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f63
-MetaChannel [context: cps-50651, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50651]
-2013-11-07 10:59:42,516-0600 INFO MetaChannel MetaChannel [context: cps-50651, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50651]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-07 10:59:42,516-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-07 10:59:42,536-0600 INFO RequestHandler Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-07 10:59:42,617-0600 INFO RequestHandler Handler(tag: 42566, SUBMITJOB) unregistering (send)
-Avg stream buf: 0
-2013-11-07 10:59:44,141-0600 INFO Multiplexer Avg stream buf: 0
-2013-11-07 10:59:44,384-0600 DEBUG Cpu Completed
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:59:44,384-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 10:59:44,384-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-Channel config: null -> {keepalive=null}
-2013-11-07 10:59:44,385-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-07 10:59:44,385-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-TCPChannel [type: server, contact: 127.0.0.1:50649] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:59:44,385-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50649] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 7545, JOBSTATUS) unregistering (send)
-2013-11-07 10:59:44,385-0600 INFO RequestHandler Handler(tag: 7545, JOBSTATUS) unregistering (send)
-2013-11-07 10:59:44,385-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-null: Channel shut down
-2013-11-07 10:59:44,385-0600 INFO AbstractTCPChannel null: Channel shut down
-grind.mcs.anl.gov:0 (quality: 107.0) submitting urn:job-2-1383759274918-1383759274919: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 10:59:44,385-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 107.0) submitting urn:job-2-1383759274918-1383759274919: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-MetaChannel [context: cps-50649, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50649]] binding to NullChannel
-2013-11-07 10:59:44,385-0600 INFO MetaChannel MetaChannel [context: cps-50649, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50649]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50649]
-2013-11-07 10:59:44,386-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50649]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:59:44,386-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 10:59:44,386-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:59:44,386-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 10:59:44,386-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:59:44,386-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 10:59:44,386-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:59:44,386-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 10:59:44,387-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:59:44,387-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 10:59:44,387-0600 INFO ChannelManager Channel exception handled
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:59:44,387-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel config: null -> {keepalive=null}
-2013-11-07 10:59:44,387-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50651] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:59:44,387-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50651] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-07 10:59:44,387-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-50651, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50651]] binding to NullChannel
-2013-11-07 10:59:44,387-0600 INFO MetaChannel MetaChannel [context: cps-50651, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50651]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50651]
-2013-11-07 10:59:44,387-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50651]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:59:44,387-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 10:59:44,387-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:59:44,387-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 10:59:44,387-0600 INFO ChannelManager Channel exception handled
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 10:59:44,391-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 10:59:44,391-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 10:59:44,392-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7546, JOBSTATUS) unregistering (send)
-2013-11-07 10:59:44,392-0600 INFO RequestHandler Handler(tag: 7546, JOBSTATUS) unregistering (send)
-Channel is closed. Removing.
-2013-11-07 10:59:44,402-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 10:59:44,402-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-07 10:59:44,402-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50649] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 10:59:44,402-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50649] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 10:59:44,402-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50649]
-2013-11-07 10:59:44,402-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50649]
-Channel is closed. Removing.
-2013-11-07 10:59:44,402-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 10:59:44,402-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-07 10:59:44,403-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50651] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 10:59:44,403-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50651] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 10:59:44,403-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50651]
-2013-11-07 10:59:44,403-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50651]
-2013-11-07 10:59:44,451-0600 DEBUG Cpu Completed
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:59:44,451-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-crush.mcs.anl.gov:0 jobTerminated
-2013-11-07 10:59:44,451-0600 INFO Cpu crush.mcs.anl.gov:0 jobTerminated
-Channel config: null -> {keepalive=null}
-2013-11-07 10:59:44,451-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-07 10:59:44,451-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-TCPChannel [type: server, contact: 127.0.0.1:50650] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:59:44,451-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50650] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 106, JOBSTATUS) unregistering (send)
-2013-11-07 10:59:44,451-0600 INFO RequestHandler Handler(tag: 106, JOBSTATUS) unregistering (send)
-2013-11-07 10:59:44,451-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-07 10:59:44,451-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-null: Channel shut down
-2013-11-07 10:59:44,451-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-50650, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50650]] binding to NullChannel
-2013-11-07 10:59:44,452-0600 INFO MetaChannel MetaChannel [context: cps-50650, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50650]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50650]
-2013-11-07 10:59:44,452-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50650]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:59:44,452-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 10:59:44,452-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:59:44,452-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 10:59:44,452-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:59:44,452-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 10:59:44,452-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:59:44,452-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 10:59:44,452-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 10:59:44,452-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 10:59:44,452-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-07 10:59:44,463-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 10:59:44,463-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-07 10:59:44,463-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50650] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 10:59:44,463-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50650] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 10:59:44,463-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50650]
-2013-11-07 10:59:44,463-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50650]
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 10:59:46,436-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 10:59:46,436-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 10:59:46,436-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 10:59:46,436-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-07 10:59:46,436-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7547, JOBSTATUS) unregistering (send)
-2013-11-07 10:59:46,436-0600 INFO RequestHandler Handler(tag: 7547, JOBSTATUS) unregistering (send)
-2013-11-07 10:59:46,436-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105749.220, now=131107-105950.070, channel=TCPChannel [type: server, contact: 127.0.0.1:50643]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:59:50,070-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105749.220, now=131107-105950.070, channel=TCPChannel [type: server, contact: 127.0.0.1:50643]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 10:59:50,070-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105749.220, now=131107-105950.070, channel=TCPChannel [type: server, contact: 127.0.0.1:50643]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:59:50,070-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105749.220, now=131107-105950.070, channel=TCPChannel [type: server, contact: 127.0.0.1:50643]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 10:59:50,070-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105749.220, now=131107-105950.070, channel=TCPChannel [type: server, contact: 127.0.0.1:50643]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:59:50,070-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105749.220, now=131107-105950.070, channel=TCPChannel [type: server, contact: 127.0.0.1:50643]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 10:59:50,070-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105749.220, now=131107-105950.070, channel=TCPChannel [type: server, contact: 127.0.0.1:50643]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:59:50,070-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105749.220, now=131107-105950.070, channel=TCPChannel [type: server, contact: 127.0.0.1:50643]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 10:59:50,070-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105749.220, now=131107-105950.070, channel=TCPChannel [type: server, contact: 127.0.0.1:50643]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:59:50,070-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105749.220, now=131107-105950.070, channel=TCPChannel [type: server, contact: 127.0.0.1:50643]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-07 10:59:50,070-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50643] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105749.220, now=131107-105950.070, channel=TCPChannel [type: server, contact: 127.0.0.1:50643]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:59:50,070-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50643] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105749.220, now=131107-105950.070, channel=TCPChannel [type: server, contact: 127.0.0.1:50643]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 10:59:50,070-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105749.422, now=131107-105950.280, channel=TCPChannel [type: server, contact: 127.0.0.1:50644]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:59:50,280-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105749.422, now=131107-105950.280, channel=TCPChannel [type: server, contact: 127.0.0.1:50644]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 10:59:50,280-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105749.422, now=131107-105950.280, channel=TCPChannel [type: server, contact: 127.0.0.1:50644]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:59:50,280-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105749.422, now=131107-105950.280, channel=TCPChannel [type: server, contact: 127.0.0.1:50644]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 10:59:50,280-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105749.422, now=131107-105950.280, channel=TCPChannel [type: server, contact: 127.0.0.1:50644]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:59:50,280-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105749.422, now=131107-105950.280, channel=TCPChannel [type: server, contact: 127.0.0.1:50644]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 10:59:50,280-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105749.422, now=131107-105950.280, channel=TCPChannel [type: server, contact: 127.0.0.1:50644]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:59:50,280-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105749.422, now=131107-105950.280, channel=TCPChannel [type: server, contact: 127.0.0.1:50644]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 10:59:50,280-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105749.422, now=131107-105950.280, channel=TCPChannel [type: server, contact: 127.0.0.1:50644]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:59:50,280-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105749.422, now=131107-105950.280, channel=TCPChannel [type: server, contact: 127.0.0.1:50644]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-07 10:59:50,280-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50644] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105749.422, now=131107-105950.280, channel=TCPChannel [type: server, contact: 127.0.0.1:50644]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:59:50,280-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50644] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105749.422, now=131107-105950.280, channel=TCPChannel [type: server, contact: 127.0.0.1:50644]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 10:59:50,280-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105749.623, now=131107-105950.480, channel=TCPChannel [type: server, contact: 127.0.0.1:50645]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:59:50,481-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105749.623, now=131107-105950.480, channel=TCPChannel [type: server, contact: 127.0.0.1:50645]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 10:59:50,481-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105749.623, now=131107-105950.480, channel=TCPChannel [type: server, contact: 127.0.0.1:50645]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:59:50,481-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105749.623, now=131107-105950.480, channel=TCPChannel [type: server, contact: 127.0.0.1:50645]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-07 10:59:50,481-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50645] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105749.623, now=131107-105950.480, channel=TCPChannel [type: server, contact: 127.0.0.1:50645]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 10:59:50,481-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50645] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105749.623, now=131107-105950.480, channel=TCPChannel [type: server, contact: 127.0.0.1:50645]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 10:59:50,481-0600 INFO ChannelManager Channel already shut down
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 10:59:51,553-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 10:59:54,150-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 10:59:59,196-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 11:00:04,158-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:00:14,166-0600 INFO Multiplexer No streams
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 11:00:16,954-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50652] for addition
-2013-11-07 11:00:16,955-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50652] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:50652]
-2013-11-07 11:00:16,955-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:50652]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f62
-2013-11-07 11:00:16,995-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f62
-MetaChannel [context: cps-50652, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50652]
-2013-11-07 11:00:16,995-0600 INFO MetaChannel MetaChannel [context: cps-50652, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50652]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-07 11:00:16,995-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-07 11:00:17,016-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-07 11:00:17,016-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-07 11:00:17,096-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-07 11:00:17,121-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 108.0) submitting urn:job-0-1383759274921-1383759274922: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 11:00:17,121-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 108.0) submitting urn:job-0-1383759274921-1383759274922: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 11:00:17,122-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-07 11:00:17,122-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-07 11:00:17,128-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7548, JOBSTATUS) unregistering (send)
-2013-11-07 11:00:17,128-0600 INFO RequestHandler Handler(tag: 7548, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 11:00:17,158-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50653] for addition
-2013-11-07 11:00:17,158-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50653] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:50653]
-2013-11-07 11:00:17,158-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:50653]
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f61
-2013-11-07 11:00:17,197-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f61
-MetaChannel [context: cps-50653, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50653]
-2013-11-07 11:00:17,197-0600 INFO MetaChannel MetaChannel [context: cps-50653, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50653]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-07 11:00:17,197-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-07 11:00:17,237-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-07 11:00:17,318-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-07 11:00:17,322-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-crush.mcs.anl.gov:0 (quality: 38.0) submitting urn:job-1-1383759274924-1383759274925: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 11:00:17,322-0600 INFO Cpu crush.mcs.anl.gov:0 (quality: 38.0) submitting urn:job-1-1383759274924-1383759274925: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 11:00:17,327-0600 DEBUG Cpu Active workerid=crush.mcs.anl.gov:000000
-Handler(tag: 107, JOBSTATUS) unregistering (send)
-2013-11-07 11:00:17,327-0600 INFO RequestHandler Handler(tag: 107, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 11:00:17,376-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50654] for addition
-2013-11-07 11:00:17,376-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50654] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:50654]
-2013-11-07 11:00:17,376-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:50654]
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f60
-2013-11-07 11:00:17,398-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f60
-MetaChannel [context: cps-50654, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50654]
-2013-11-07 11:00:17,399-0600 INFO MetaChannel MetaChannel [context: cps-50654, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50654]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-07 11:00:17,399-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-07 11:00:17,439-0600 INFO RequestHandler Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-07 11:00:17,520-0600 INFO RequestHandler Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-07 11:00:19,176-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 11:00:19,176-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 11:00:19,176-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:00:19,176-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel config: null -> {keepalive=null}
-2013-11-07 11:00:19,176-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-07 11:00:19,176-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-Handler(tag: 7549, JOBSTATUS) unregistering (send)
-2013-11-07 11:00:19,176-0600 INFO RequestHandler Handler(tag: 7549, JOBSTATUS) unregistering (send)
-TCPChannel [type: server, contact: 127.0.0.1:50652] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:00:19,177-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50652] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-grind.mcs.anl.gov:0 (quality: 109.0) submitting urn:job-2-1383759274927-1383759274928: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 11:00:19,177-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 109.0) submitting urn:job-2-1383759274927-1383759274928: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-null: Channel shut down
-2013-11-07 11:00:19,177-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-50652, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50652]] binding to NullChannel
-2013-11-07 11:00:19,178-0600 INFO MetaChannel MetaChannel [context: cps-50652, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50652]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50652]
-2013-11-07 11:00:19,178-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50652]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:00:19,178-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 11:00:19,178-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:00:19,178-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 11:00:19,179-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:00:19,179-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 11:00:19,179-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:00:19,179-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 11:00:19,179-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:00:19,179-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 11:00:19,179-0600 INFO ChannelManager Channel exception handled
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:00:19,180-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel config: null -> {keepalive=null}
-2013-11-07 11:00:19,180-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50654] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:00:19,180-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50654] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-07 11:00:19,180-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-50654, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50654]] binding to NullChannel
-2013-11-07 11:00:19,180-0600 INFO MetaChannel MetaChannel [context: cps-50654, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50654]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50654]
-2013-11-07 11:00:19,180-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50654]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:00:19,180-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 11:00:19,180-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:00:19,180-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 11:00:19,180-0600 INFO ChannelManager Channel exception handled
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 11:00:19,184-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 11:00:19,184-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 11:00:19,184-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7550, JOBSTATUS) unregistering (send)
-2013-11-07 11:00:19,184-0600 INFO RequestHandler Handler(tag: 7550, JOBSTATUS) unregistering (send)
-Channel is closed. Removing.
-2013-11-07 11:00:19,184-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 11:00:19,185-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-07 11:00:19,185-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50652] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 11:00:19,185-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50652] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 11:00:19,185-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50652]
-2013-11-07 11:00:19,185-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50652]
-Channel is closed. Removing.
-2013-11-07 11:00:19,185-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 11:00:19,185-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-07 11:00:19,185-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50654] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 11:00:19,185-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50654] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 11:00:19,185-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50654]
-2013-11-07 11:00:19,185-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50654]
-2013-11-07 11:00:19,341-0600 DEBUG Cpu Completed
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:00:19,341-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-crush.mcs.anl.gov:0 jobTerminated
-2013-11-07 11:00:19,341-0600 INFO Cpu crush.mcs.anl.gov:0 jobTerminated
-Channel config: null -> {keepalive=null}
-2013-11-07 11:00:19,342-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-07 11:00:19,342-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-TCPChannel [type: server, contact: 127.0.0.1:50653] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:00:19,342-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50653] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 108, JOBSTATUS) unregistering (send)
-2013-11-07 11:00:19,342-0600 INFO RequestHandler Handler(tag: 108, JOBSTATUS) unregistering (send)
-2013-11-07 11:00:19,342-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-null: Channel shut down
-2013-11-07 11:00:19,342-0600 INFO AbstractTCPChannel null: Channel shut down
-2013-11-07 11:00:19,342-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-MetaChannel [context: cps-50653, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50653]] binding to NullChannel
-2013-11-07 11:00:19,342-0600 INFO MetaChannel MetaChannel [context: cps-50653, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50653]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50653]
-2013-11-07 11:00:19,342-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50653]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:00:19,342-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 11:00:19,342-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:00:19,342-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 11:00:19,343-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:00:19,343-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 11:00:19,343-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:00:19,343-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 11:00:19,343-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:00:19,343-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 11:00:19,343-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-07 11:00:19,346-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 11:00:19,346-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-07 11:00:19,346-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50653] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 11:00:19,346-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50653] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 11:00:19,346-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50653]
-2013-11-07 11:00:19,346-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50653]
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 11:00:21,198-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 11:00:21,198-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 11:00:21,198-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 11:00:21,198-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-07 11:00:21,198-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7551, JOBSTATUS) unregistering (send)
-2013-11-07 11:00:21,198-0600 INFO RequestHandler Handler(tag: 7551, JOBSTATUS) unregistering (send)
-2013-11-07 11:00:21,198-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-No streams
-2013-11-07 11:00:24,180-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:00:28,871-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:00:34,188-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:00:35,669-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105838.161, now=131107-110039.006, channel=TCPChannel [type: server, contact: 127.0.0.1:50646]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:00:39,006-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105838.161, now=131107-110039.006, channel=TCPChannel [type: server, contact: 127.0.0.1:50646]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 11:00:39,006-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105838.161, now=131107-110039.006, channel=TCPChannel [type: server, contact: 127.0.0.1:50646]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:00:39,006-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105838.161, now=131107-110039.006, channel=TCPChannel [type: server, contact: 127.0.0.1:50646]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 11:00:39,006-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105838.161, now=131107-110039.006, channel=TCPChannel [type: server, contact: 127.0.0.1:50646]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:00:39,006-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105838.161, now=131107-110039.006, channel=TCPChannel [type: server, contact: 127.0.0.1:50646]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 11:00:39,006-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105838.161, now=131107-110039.006, channel=TCPChannel [type: server, contact: 127.0.0.1:50646]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:00:39,006-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105838.161, now=131107-110039.006, channel=TCPChannel [type: server, contact: 127.0.0.1:50646]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 11:00:39,006-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105838.161, now=131107-110039.006, channel=TCPChannel [type: server, contact: 127.0.0.1:50646]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:00:39,006-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105838.161, now=131107-110039.006, channel=TCPChannel [type: server, contact: 127.0.0.1:50646]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-07 11:00:39,006-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50646] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105838.161, now=131107-110039.006, channel=TCPChannel [type: server, contact: 127.0.0.1:50646]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:00:39,006-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50646] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105838.161, now=131107-110039.006, channel=TCPChannel [type: server, contact: 127.0.0.1:50646]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 11:00:39,007-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105838.368, now=131107-110039.220, channel=TCPChannel [type: server, contact: 127.0.0.1:50647]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:00:39,220-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105838.368, now=131107-110039.220, channel=TCPChannel [type: server, contact: 127.0.0.1:50647]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 11:00:39,220-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105838.368, now=131107-110039.220, channel=TCPChannel [type: server, contact: 127.0.0.1:50647]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:00:39,220-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105838.368, now=131107-110039.220, channel=TCPChannel [type: server, contact: 127.0.0.1:50647]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 11:00:39,220-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105838.368, now=131107-110039.220, channel=TCPChannel [type: server, contact: 127.0.0.1:50647]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:00:39,220-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105838.368, now=131107-110039.220, channel=TCPChannel [type: server, contact: 127.0.0.1:50647]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 11:00:39,220-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105838.368, now=131107-110039.220, channel=TCPChannel [type: server, contact: 127.0.0.1:50647]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:00:39,220-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105838.368, now=131107-110039.220, channel=TCPChannel [type: server, contact: 127.0.0.1:50647]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 11:00:39,220-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105838.368, now=131107-110039.220, channel=TCPChannel [type: server, contact: 127.0.0.1:50647]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:00:39,220-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105838.368, now=131107-110039.220, channel=TCPChannel [type: server, contact: 127.0.0.1:50647]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-07 11:00:39,220-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50647] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105838.368, now=131107-110039.220, channel=TCPChannel [type: server, contact: 127.0.0.1:50647]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:00:39,220-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50647] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105838.368, now=131107-110039.220, channel=TCPChannel [type: server, contact: 127.0.0.1:50647]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 11:00:39,220-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105838.569, now=131107-110039.430, channel=TCPChannel [type: server, contact: 127.0.0.1:50648]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:00:39,430-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105838.569, now=131107-110039.430, channel=TCPChannel [type: server, contact: 127.0.0.1:50648]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 11:00:39,430-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105838.569, now=131107-110039.430, channel=TCPChannel [type: server, contact: 127.0.0.1:50648]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:00:39,430-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105838.569, now=131107-110039.430, channel=TCPChannel [type: server, contact: 127.0.0.1:50648]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-07 11:00:39,430-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50648] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105838.569, now=131107-110039.430, channel=TCPChannel [type: server, contact: 127.0.0.1:50648]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:00:39,430-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50648] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105838.569, now=131107-110039.430, channel=TCPChannel [type: server, contact: 127.0.0.1:50648]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 11:00:39,430-0600 INFO ChannelManager Channel already shut down
-No streams
-2013-11-07 11:00:44,196-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:00:54,204-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:01:04,211-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 11:01:06,189-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:01:12,143-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:01:14,219-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:01:24,227-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:01:34,235-0600 INFO Multiplexer No streams
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105942.193, now=131107-110143.039, channel=TCPChannel [type: server, contact: 127.0.0.1:50649]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:01:43,039-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105942.193, now=131107-110143.039, channel=TCPChannel [type: server, contact: 127.0.0.1:50649]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 11:01:43,039-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105942.193, now=131107-110143.039, channel=TCPChannel [type: server, contact: 127.0.0.1:50649]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:01:43,039-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105942.193, now=131107-110143.039, channel=TCPChannel [type: server, contact: 127.0.0.1:50649]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 11:01:43,039-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105942.193, now=131107-110143.039, channel=TCPChannel [type: server, contact: 127.0.0.1:50649]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:01:43,039-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105942.193, now=131107-110143.039, channel=TCPChannel [type: server, contact: 127.0.0.1:50649]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 11:01:43,039-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105942.193, now=131107-110143.039, channel=TCPChannel [type: server, contact: 127.0.0.1:50649]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:01:43,039-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105942.193, now=131107-110143.039, channel=TCPChannel [type: server, contact: 127.0.0.1:50649]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 11:01:43,039-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105942.193, now=131107-110143.039, channel=TCPChannel [type: server, contact: 127.0.0.1:50649]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:01:43,039-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105942.193, now=131107-110143.039, channel=TCPChannel [type: server, contact: 127.0.0.1:50649]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-07 11:01:43,039-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50649] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105942.193, now=131107-110143.039, channel=TCPChannel [type: server, contact: 127.0.0.1:50649]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:01:43,039-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50649] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105942.193, now=131107-110143.039, channel=TCPChannel [type: server, contact: 127.0.0.1:50649]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 11:01:43,039-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105942.415, now=131107-110143.252, channel=TCPChannel [type: server, contact: 127.0.0.1:50650]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:01:43,252-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105942.415, now=131107-110143.252, channel=TCPChannel [type: server, contact: 127.0.0.1:50650]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 11:01:43,252-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105942.415, now=131107-110143.252, channel=TCPChannel [type: server, contact: 127.0.0.1:50650]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:01:43,252-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105942.415, now=131107-110143.252, channel=TCPChannel [type: server, contact: 127.0.0.1:50650]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 11:01:43,253-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105942.415, now=131107-110143.252, channel=TCPChannel [type: server, contact: 127.0.0.1:50650]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:01:43,253-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105942.415, now=131107-110143.252, channel=TCPChannel [type: server, contact: 127.0.0.1:50650]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 11:01:43,253-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105942.415, now=131107-110143.252, channel=TCPChannel [type: server, contact: 127.0.0.1:50650]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:01:43,253-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105942.415, now=131107-110143.252, channel=TCPChannel [type: server, contact: 127.0.0.1:50650]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 11:01:43,253-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105942.415, now=131107-110143.252, channel=TCPChannel [type: server, contact: 127.0.0.1:50650]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:01:43,253-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105942.415, now=131107-110143.252, channel=TCPChannel [type: server, contact: 127.0.0.1:50650]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-07 11:01:43,253-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50650] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105942.415, now=131107-110143.252, channel=TCPChannel [type: server, contact: 127.0.0.1:50650]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:01:43,253-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50650] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105942.415, now=131107-110143.252, channel=TCPChannel [type: server, contact: 127.0.0.1:50650]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 11:01:43,253-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105942.617, now=131107-110143.476, channel=TCPChannel [type: server, contact: 127.0.0.1:50651]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:01:43,476-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105942.617, now=131107-110143.476, channel=TCPChannel [type: server, contact: 127.0.0.1:50651]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 11:01:43,476-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105942.617, now=131107-110143.476, channel=TCPChannel [type: server, contact: 127.0.0.1:50651]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:01:43,476-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105942.617, now=131107-110143.476, channel=TCPChannel [type: server, contact: 127.0.0.1:50651]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-07 11:01:43,476-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50651] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105942.617, now=131107-110143.476, channel=TCPChannel [type: server, contact: 127.0.0.1:50651]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:01:43,476-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50651] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-105942.617, now=131107-110143.476, channel=TCPChannel [type: server, contact: 127.0.0.1:50651]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 11:01:43,476-0600 INFO ChannelManager Channel already shut down
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 11:01:43,507-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 11:01:44,243-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:01:48,617-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:01:54,251-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:02:04,259-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:02:14,267-0600 INFO Multiplexer No streams
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110017.096, now=131107-110217.956, channel=TCPChannel [type: server, contact: 127.0.0.1:50652]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:02:17,956-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110017.096, now=131107-110217.956, channel=TCPChannel [type: server, contact: 127.0.0.1:50652]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 11:02:17,956-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110017.096, now=131107-110217.956, channel=TCPChannel [type: server, contact: 127.0.0.1:50652]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:02:17,956-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110017.096, now=131107-110217.956, channel=TCPChannel [type: server, contact: 127.0.0.1:50652]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 11:02:17,956-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110017.096, now=131107-110217.956, channel=TCPChannel [type: server, contact: 127.0.0.1:50652]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:02:17,956-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110017.096, now=131107-110217.956, channel=TCPChannel [type: server, contact: 127.0.0.1:50652]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 11:02:17,956-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110017.096, now=131107-110217.956, channel=TCPChannel [type: server, contact: 127.0.0.1:50652]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:02:17,956-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110017.096, now=131107-110217.956, channel=TCPChannel [type: server, contact: 127.0.0.1:50652]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 11:02:17,956-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110017.096, now=131107-110217.956, channel=TCPChannel [type: server, contact: 127.0.0.1:50652]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:02:17,956-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110017.096, now=131107-110217.956, channel=TCPChannel [type: server, contact: 127.0.0.1:50652]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-07 11:02:17,956-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50652] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110017.096, now=131107-110217.956, channel=TCPChannel [type: server, contact: 127.0.0.1:50652]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:02:17,956-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50652] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110017.096, now=131107-110217.956, channel=TCPChannel [type: server, contact: 127.0.0.1:50652]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 11:02:17,956-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110017.318, now=131107-110218.158, channel=TCPChannel [type: server, contact: 127.0.0.1:50653]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:02:18,158-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110017.318, now=131107-110218.158, channel=TCPChannel [type: server, contact: 127.0.0.1:50653]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 11:02:18,158-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110017.318, now=131107-110218.158, channel=TCPChannel [type: server, contact: 127.0.0.1:50653]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:02:18,158-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110017.318, now=131107-110218.158, channel=TCPChannel [type: server, contact: 127.0.0.1:50653]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 11:02:18,158-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110017.318, now=131107-110218.158, channel=TCPChannel [type: server, contact: 127.0.0.1:50653]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:02:18,158-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110017.318, now=131107-110218.158, channel=TCPChannel [type: server, contact: 127.0.0.1:50653]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 11:02:18,159-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110017.318, now=131107-110218.158, channel=TCPChannel [type: server, contact: 127.0.0.1:50653]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:02:18,159-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110017.318, now=131107-110218.158, channel=TCPChannel [type: server, contact: 127.0.0.1:50653]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 11:02:18,159-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110017.318, now=131107-110218.158, channel=TCPChannel [type: server, contact: 127.0.0.1:50653]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:02:18,159-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110017.318, now=131107-110218.158, channel=TCPChannel [type: server, contact: 127.0.0.1:50653]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-07 11:02:18,159-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50653] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110017.318, now=131107-110218.158, channel=TCPChannel [type: server, contact: 127.0.0.1:50653]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:02:18,159-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50653] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110017.318, now=131107-110218.158, channel=TCPChannel [type: server, contact: 127.0.0.1:50653]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 11:02:18,159-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110017.519, now=131107-110218.376, channel=TCPChannel [type: server, contact: 127.0.0.1:50654]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:02:18,376-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110017.519, now=131107-110218.376, channel=TCPChannel [type: server, contact: 127.0.0.1:50654]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 11:02:18,376-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110017.519, now=131107-110218.376, channel=TCPChannel [type: server, contact: 127.0.0.1:50654]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:02:18,376-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110017.519, now=131107-110218.376, channel=TCPChannel [type: server, contact: 127.0.0.1:50654]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-07 11:02:18,376-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50654] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110017.519, now=131107-110218.376, channel=TCPChannel [type: server, contact: 127.0.0.1:50654]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:02:18,376-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50654] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110017.519, now=131107-110218.376, channel=TCPChannel [type: server, contact: 127.0.0.1:50654]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 11:02:18,376-0600 INFO ChannelManager Channel already shut down
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-07 11:02:20,826-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-07 11:02:24,275-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:02:25,091-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:02:34,283-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:02:44,290-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:02:54,298-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:02:58,144-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:03:01,565-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:03:04,306-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:03:14,314-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:03:24,322-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:03:34,329-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:03:35,462-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:03:38,039-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:03:44,337-0600 INFO Multiplexer No streams
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 11:03:51,524-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50655] for addition
-2013-11-07 11:03:51,525-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50655] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:50655]
-2013-11-07 11:03:51,525-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:50655]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f5f
-2013-11-07 11:03:51,558-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f5f
-MetaChannel [context: cps-50655, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50655]
-2013-11-07 11:03:51,558-0600 INFO MetaChannel MetaChannel [context: cps-50655, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50655]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-07 11:03:51,558-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-07 11:03:51,619-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-07 11:03:51,619-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-07 11:03:51,699-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-07 11:03:51,737-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 110.0) submitting urn:job-0-1383759274930-1383759274931: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 11:03:51,737-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 110.0) submitting urn:job-0-1383759274930-1383759274931: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 11:03:51,738-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-07 11:03:51,739-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-07 11:03:51,743-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7552, JOBSTATUS) unregistering (send)
-2013-11-07 11:03:51,743-0600 INFO RequestHandler Handler(tag: 7552, JOBSTATUS) unregistering (send)
-2013-11-07 11:03:53,937-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 11:03:53,937-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 11:03:53,937-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Channel IOException
-java.net.SocketException: Connection reset
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:129)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:03:53,938-0600 INFO Sender Channel IOException
-java.net.SocketException: Connection reset
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:129)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 7553, JOBSTATUS) unregistering (send)
-2013-11-07 11:03:53,938-0600 INFO RequestHandler Handler(tag: 7553, JOBSTATUS) unregistering (send)
-2013-11-07 11:03:53,938-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Channel config: null -> {keepalive=null}
-2013-11-07 11:03:53,938-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50655] handling channel exception
-java.net.SocketException: Connection reset
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:129)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:03:53,938-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50655] handling channel exception
-java.net.SocketException: Connection reset
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:129)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:03:53,938-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-null: Channel shut down
-2013-11-07 11:03:53,938-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-50655, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50655]] binding to NullChannel
-2013-11-07 11:03:53,938-0600 INFO MetaChannel MetaChannel [context: cps-50655, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50655]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50655]
-2013-11-07 11:03:53,939-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50655]
-Notifying commands and handlers about exception
-java.net.SocketException: Connection reset
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:129)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:03:53,939-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Connection reset
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:129)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 11:03:53,939-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Connection reset
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:129)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:03:53,939-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Connection reset
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:129)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 11:03:53,939-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Connection reset
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:129)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:03:53,939-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Connection reset
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:129)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 11:03:53,939-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Connection reset
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:129)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:03:53,939-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Connection reset
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:129)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 11:03:53,939-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Connection reset
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:129)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:03:53,939-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Connection reset
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:129)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 11:03:53,939-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-07 11:03:53,946-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 11:03:53,946-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-07 11:03:53,946-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50655] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 11:03:53,946-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50655] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 11:03:53,946-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50655]
-2013-11-07 11:03:53,946-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50655]
-No streams
-2013-11-07 11:03:54,347-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:04:04,355-0600 INFO Multiplexer No streams
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 11:04:07,236-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50656] for addition
-2013-11-07 11:04:07,236-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50656] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:50656]
-2013-11-07 11:04:07,236-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:50656]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f5e
-2013-11-07 11:04:07,264-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f5e
-MetaChannel [context: cps-50656, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50656]
-2013-11-07 11:04:07,264-0600 INFO MetaChannel MetaChannel [context: cps-50656, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50656]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-07 11:04:07,264-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-07 11:04:07,305-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-07 11:04:07,305-0600 INFO Sender Sender TCPChannel queue size: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 11:04:12,780-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-07 11:04:14,365-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:04:14,513-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-07 11:04:24,373-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-07 11:04:34,382-0600 INFO Multiplexer Avg stream buf: 0
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 11:04:34,698-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50657] for addition
-2013-11-07 11:04:34,698-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50657] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:50657]
-2013-11-07 11:04:34,698-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:50657]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f5d
-2013-11-07 11:04:34,724-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f5d
-MetaChannel [context: cps-50657, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50657]
-2013-11-07 11:04:34,724-0600 INFO MetaChannel MetaChannel [context: cps-50657, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50657]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-07 11:04:34,725-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-07 11:04:34,765-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-07 11:04:34,765-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-07 11:04:34,845-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-07 11:04:34,885-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 111.0) submitting urn:job-0-1383759274933-1383759274934: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 11:04:34,885-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 111.0) submitting urn:job-0-1383759274933-1383759274934: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 11:04:34,886-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-07 11:04:34,886-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-07 11:04:34,891-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7554, JOBSTATUS) unregistering (send)
-2013-11-07 11:04:34,892-0600 INFO RequestHandler Handler(tag: 7554, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 11:04:34,904-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50658] for addition
-2013-11-07 11:04:34,904-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50658] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:50658]
-2013-11-07 11:04:34,904-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:50658]
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f5c
-2013-11-07 11:04:34,926-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f5c
-MetaChannel [context: cps-50658, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50658]
-2013-11-07 11:04:34,926-0600 INFO MetaChannel MetaChannel [context: cps-50658, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50658]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-07 11:04:34,926-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-07 11:04:34,967-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-07 11:04:35,047-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-07 11:04:35,086-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-crush.mcs.anl.gov:0 (quality: 39.0) submitting urn:job-1-1383759274936-1383759274937: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 11:04:35,086-0600 INFO Cpu crush.mcs.anl.gov:0 (quality: 39.0) submitting urn:job-1-1383759274936-1383759274937: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 11:04:35,092-0600 DEBUG Cpu Active workerid=crush.mcs.anl.gov:000000
-Handler(tag: 109, JOBSTATUS) unregistering (send)
-2013-11-07 11:04:35,092-0600 INFO RequestHandler Handler(tag: 109, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 11:04:35,107-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50659] for addition
-2013-11-07 11:04:35,107-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50659] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:50659]
-2013-11-07 11:04:35,107-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:50659]
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f5b
-2013-11-07 11:04:35,148-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f5b
-MetaChannel [context: cps-50659, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50659]
-2013-11-07 11:04:35,148-0600 INFO MetaChannel MetaChannel [context: cps-50659, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50659]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-07 11:04:35,148-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-07 11:04:35,188-0600 INFO RequestHandler Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-07 11:04:35,269-0600 INFO RequestHandler Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-07 11:04:36,925-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 11:04:36,925-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 11:04:36,925-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:04:36,925-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 7555, JOBSTATUS) unregistering (send)
-2013-11-07 11:04:36,925-0600 INFO RequestHandler Handler(tag: 7555, JOBSTATUS) unregistering (send)
-2013-11-07 11:04:36,925-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-Channel config: null -> {keepalive=null}
-2013-11-07 11:04:36,925-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50657] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:04:36,925-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50657] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-07 11:04:36,925-0600 INFO AbstractTCPChannel null: Channel shut down
-grind.mcs.anl.gov:0 (quality: 112.0) submitting urn:job-2-1383759274939-1383759274940: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 11:04:36,925-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 112.0) submitting urn:job-2-1383759274939-1383759274940: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-MetaChannel [context: cps-50657, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50657]] binding to NullChannel
-2013-11-07 11:04:36,925-0600 INFO MetaChannel MetaChannel [context: cps-50657, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50657]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50657]
-2013-11-07 11:04:36,926-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50657]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:04:36,926-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 11:04:36,926-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:04:36,926-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 11:04:36,926-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:04:36,926-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 11:04:36,926-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:04:36,926-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 11:04:36,926-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:04:36,926-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 11:04:36,926-0600 INFO ChannelManager Channel exception handled
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:04:36,926-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel config: null -> {keepalive=null}
-2013-11-07 11:04:36,927-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50659] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:04:36,927-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50659] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-07 11:04:36,927-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-50659, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50659]] binding to NullChannel
-2013-11-07 11:04:36,927-0600 INFO MetaChannel MetaChannel [context: cps-50659, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50659]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50659]
-2013-11-07 11:04:36,927-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50659]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:04:36,927-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 11:04:36,927-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:04:36,927-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 11:04:36,927-0600 INFO ChannelManager Channel exception handled
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 11:04:36,930-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 11:04:36,931-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 11:04:36,931-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7556, JOBSTATUS) unregistering (send)
-2013-11-07 11:04:36,931-0600 INFO RequestHandler Handler(tag: 7556, JOBSTATUS) unregistering (send)
-Channel is closed. Removing.
-2013-11-07 11:04:36,937-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 11:04:36,938-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-07 11:04:36,938-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50659] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 11:04:36,938-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50659] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 11:04:36,938-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50659]
-2013-11-07 11:04:36,938-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50659]
-Channel is closed. Removing.
-2013-11-07 11:04:36,938-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 11:04:36,938-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-07 11:04:36,938-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50657] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 11:04:36,938-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50657] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 11:04:36,938-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50657]
-2013-11-07 11:04:36,938-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50657]
-2013-11-07 11:04:37,276-0600 DEBUG Cpu Completed
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:04:37,276-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-crush.mcs.anl.gov:0 jobTerminated
-2013-11-07 11:04:37,276-0600 INFO Cpu crush.mcs.anl.gov:0 jobTerminated
-Channel config: null -> {keepalive=null}
-2013-11-07 11:04:37,277-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-07 11:04:37,277-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-TCPChannel [type: server, contact: 127.0.0.1:50658] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:04:37,277-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50658] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:04:37,277-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-null: Channel shut down
-2013-11-07 11:04:37,277-0600 INFO AbstractTCPChannel null: Channel shut down
-2013-11-07 11:04:37,277-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-MetaChannel [context: cps-50658, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50658]] binding to NullChannel
-2013-11-07 11:04:37,277-0600 INFO MetaChannel MetaChannel [context: cps-50658, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50658]] binding to NullChannel
-Handler(tag: 110, JOBSTATUS) unregistering (send)
-2013-11-07 11:04:37,277-0600 INFO RequestHandler Handler(tag: 110, JOBSTATUS) unregistering (send)
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50658]
-2013-11-07 11:04:37,277-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50658]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:04:37,277-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 11:04:37,278-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:04:37,278-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 11:04:37,278-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:04:37,278-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 11:04:37,278-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:04:37,278-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 11:04:37,278-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:04:37,278-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 11:04:37,278-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-07 11:04:37,279-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 11:04:37,279-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-07 11:04:37,279-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50658] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 11:04:37,279-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50658] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 11:04:37,279-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50658]
-2013-11-07 11:04:37,279-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50658]
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 11:04:38,948-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 11:04:38,948-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 11:04:38,948-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 11:04:38,948-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-07 11:04:38,948-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7557, JOBSTATUS) unregistering (send)
-2013-11-07 11:04:38,948-0600 INFO RequestHandler Handler(tag: 7557, JOBSTATUS) unregistering (send)
-2013-11-07 11:04:38,948-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-Avg stream buf: 0
-2013-11-07 11:04:44,399-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:04:50,098-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 11:04:50,987-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-Avg stream buf: 0
-2013-11-07 11:04:54,407-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-07 11:05:04,415-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-07 11:05:14,424-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-07 11:05:24,432-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 11:05:27,417-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:05:27,463-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-07 11:05:34,440-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-07 11:05:44,447-0600 INFO Multiplexer Avg stream buf: 0
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110351.699, now=131107-110552.524, channel=TCPChannel [type: server, contact: 127.0.0.1:50655]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:05:52,524-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110351.699, now=131107-110552.524, channel=TCPChannel [type: server, contact: 127.0.0.1:50655]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 11:05:52,524-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110351.699, now=131107-110552.524, channel=TCPChannel [type: server, contact: 127.0.0.1:50655]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:05:52,524-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110351.699, now=131107-110552.524, channel=TCPChannel [type: server, contact: 127.0.0.1:50655]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 11:05:52,524-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110351.699, now=131107-110552.524, channel=TCPChannel [type: server, contact: 127.0.0.1:50655]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:05:52,524-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110351.699, now=131107-110552.524, channel=TCPChannel [type: server, contact: 127.0.0.1:50655]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 11:05:52,524-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110351.699, now=131107-110552.524, channel=TCPChannel [type: server, contact: 127.0.0.1:50655]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:05:52,524-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110351.699, now=131107-110552.524, channel=TCPChannel [type: server, contact: 127.0.0.1:50655]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 11:05:52,524-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110351.699, now=131107-110552.524, channel=TCPChannel [type: server, contact: 127.0.0.1:50655]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:05:52,524-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110351.699, now=131107-110552.524, channel=TCPChannel [type: server, contact: 127.0.0.1:50655]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-07 11:05:52,524-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50655] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110351.699, now=131107-110552.524, channel=TCPChannel [type: server, contact: 127.0.0.1:50655]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:05:52,524-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50655] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110351.699, now=131107-110552.524, channel=TCPChannel [type: server, contact: 127.0.0.1:50655]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 11:05:52,524-0600 INFO ChannelManager Channel already shut down
-Avg stream buf: 0
-2013-11-07 11:05:54,455-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 11:06:03,938-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-Avg stream buf: 0
-2013-11-07 11:06:04,463-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:06:04,734-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110407.304, now=131107-110608.236, channel=TCPChannel [type: server, contact: 127.0.0.1:50656]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:06:08,236-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110407.304, now=131107-110608.236, channel=TCPChannel [type: server, contact: 127.0.0.1:50656]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-07 11:06:08,236-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50656] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110407.304, now=131107-110608.236, channel=TCPChannel [type: server, contact: 127.0.0.1:50656]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:06:08,236-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50656] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110407.304, now=131107-110608.236, channel=TCPChannel [type: server, contact: 127.0.0.1:50656]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-null: Channel shut down
-2013-11-07 11:06:08,237-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-50656, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50656]] binding to NullChannel
-2013-11-07 11:06:08,237-0600 INFO MetaChannel MetaChannel [context: cps-50656, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50656]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50656]
-2013-11-07 11:06:08,237-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50656]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110407.304, now=131107-110608.236, channel=TCPChannel [type: server, contact: 127.0.0.1:50656]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:06:08,237-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110407.304, now=131107-110608.236, channel=TCPChannel [type: server, contact: 127.0.0.1:50656]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel exception handled
-2013-11-07 11:06:08,237-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-07 11:06:08,254-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 11:06:08,254-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-07 11:06:08,254-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50656] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 11:06:08,254-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50656] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 11:06:08,254-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50656]
-2013-11-07 11:06:08,254-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50656]
-No streams
-2013-11-07 11:06:14,471-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:06:24,478-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:06:34,486-0600 INFO Multiplexer No streams
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110434.845, now=131107-110635.698, channel=TCPChannel [type: server, contact: 127.0.0.1:50657]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:06:35,698-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110434.845, now=131107-110635.698, channel=TCPChannel [type: server, contact: 127.0.0.1:50657]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 11:06:35,698-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110434.845, now=131107-110635.698, channel=TCPChannel [type: server, contact: 127.0.0.1:50657]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:06:35,698-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110434.845, now=131107-110635.698, channel=TCPChannel [type: server, contact: 127.0.0.1:50657]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 11:06:35,698-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110434.845, now=131107-110635.698, channel=TCPChannel [type: server, contact: 127.0.0.1:50657]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:06:35,698-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110434.845, now=131107-110635.698, channel=TCPChannel [type: server, contact: 127.0.0.1:50657]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 11:06:35,698-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110434.845, now=131107-110635.698, channel=TCPChannel [type: server, contact: 127.0.0.1:50657]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:06:35,698-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110434.845, now=131107-110635.698, channel=TCPChannel [type: server, contact: 127.0.0.1:50657]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 11:06:35,698-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110434.845, now=131107-110635.698, channel=TCPChannel [type: server, contact: 127.0.0.1:50657]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:06:35,698-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110434.845, now=131107-110635.698, channel=TCPChannel [type: server, contact: 127.0.0.1:50657]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-07 11:06:35,698-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50657] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110434.845, now=131107-110635.698, channel=TCPChannel [type: server, contact: 127.0.0.1:50657]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:06:35,698-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50657] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110434.845, now=131107-110635.698, channel=TCPChannel [type: server, contact: 127.0.0.1:50657]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 11:06:35,698-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110435.047, now=131107-110635.905, channel=TCPChannel [type: server, contact: 127.0.0.1:50658]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:06:35,905-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110435.047, now=131107-110635.905, channel=TCPChannel [type: server, contact: 127.0.0.1:50658]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 11:06:35,905-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110435.047, now=131107-110635.905, channel=TCPChannel [type: server, contact: 127.0.0.1:50658]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:06:35,905-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110435.047, now=131107-110635.905, channel=TCPChannel [type: server, contact: 127.0.0.1:50658]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 11:06:35,905-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110435.047, now=131107-110635.905, channel=TCPChannel [type: server, contact: 127.0.0.1:50658]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:06:35,905-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110435.047, now=131107-110635.905, channel=TCPChannel [type: server, contact: 127.0.0.1:50658]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 11:06:35,905-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110435.047, now=131107-110635.905, channel=TCPChannel [type: server, contact: 127.0.0.1:50658]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:06:35,905-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110435.047, now=131107-110635.905, channel=TCPChannel [type: server, contact: 127.0.0.1:50658]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 11:06:35,905-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110435.047, now=131107-110635.905, channel=TCPChannel [type: server, contact: 127.0.0.1:50658]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:06:35,905-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110435.047, now=131107-110635.905, channel=TCPChannel [type: server, contact: 127.0.0.1:50658]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-07 11:06:35,905-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50658] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110435.047, now=131107-110635.905, channel=TCPChannel [type: server, contact: 127.0.0.1:50658]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:06:35,905-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50658] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110435.047, now=131107-110635.905, channel=TCPChannel [type: server, contact: 127.0.0.1:50658]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 11:06:35,905-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110435.269, now=131107-110636.107, channel=TCPChannel [type: server, contact: 127.0.0.1:50659]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:06:36,107-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110435.269, now=131107-110636.107, channel=TCPChannel [type: server, contact: 127.0.0.1:50659]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 11:06:36,107-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110435.269, now=131107-110636.107, channel=TCPChannel [type: server, contact: 127.0.0.1:50659]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:06:36,107-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110435.269, now=131107-110636.107, channel=TCPChannel [type: server, contact: 127.0.0.1:50659]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-07 11:06:36,107-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50659] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110435.269, now=131107-110636.107, channel=TCPChannel [type: server, contact: 127.0.0.1:50659]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:06:36,108-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50659] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110435.269, now=131107-110636.107, channel=TCPChannel [type: server, contact: 127.0.0.1:50659]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 11:06:36,108-0600 INFO ChannelManager Channel already shut down
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:06:40,411-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-07 11:06:42,052-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-07 11:06:44,494-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:06:54,502-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:07:04,510-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:07:14,518-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:07:16,885-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 11:07:19,371-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 11:07:24,526-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:07:34,534-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:07:44,542-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:07:53,359-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:07:54,550-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:07:56,689-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:08:04,558-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:08:14,565-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:08:24,573-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:08:29,833-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 11:08:34,007-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 11:08:34,581-0600 INFO Multiplexer No streams
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 11:08:44,131-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50660] for addition
-2013-11-07 11:08:44,132-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50660] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:50660]
-2013-11-07 11:08:44,132-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:50660]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f5a
-2013-11-07 11:08:44,168-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f5a
-MetaChannel [context: cps-50660, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50660]
-2013-11-07 11:08:44,168-0600 INFO MetaChannel MetaChannel [context: cps-50660, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50660]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-07 11:08:44,168-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-07 11:08:44,208-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-07 11:08:44,208-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-07 11:08:44,289-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-07 11:08:44,326-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 113.0) submitting urn:job-0-1383759274942-1383759274943: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 11:08:44,326-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 113.0) submitting urn:job-0-1383759274942-1383759274943: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 11:08:44,327-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-07 11:08:44,327-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-07 11:08:44,332-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7558, JOBSTATUS) unregistering (send)
-2013-11-07 11:08:44,332-0600 INFO RequestHandler Handler(tag: 7558, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 11:08:44,348-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50661] for addition
-2013-11-07 11:08:44,348-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50661] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:50661]
-2013-11-07 11:08:44,348-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:50661]
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f59
-2013-11-07 11:08:44,370-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f59
-MetaChannel [context: cps-50661, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50661]
-2013-11-07 11:08:44,370-0600 INFO MetaChannel MetaChannel [context: cps-50661, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50661]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-07 11:08:44,370-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-07 11:08:44,410-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-07 11:08:44,490-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-07 11:08:44,527-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-crush.mcs.anl.gov:0 (quality: 40.0) submitting urn:job-1-1383759274945-1383759274946: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 11:08:44,527-0600 INFO Cpu crush.mcs.anl.gov:0 (quality: 40.0) submitting urn:job-1-1383759274945-1383759274946: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 11:08:44,533-0600 DEBUG Cpu Active workerid=crush.mcs.anl.gov:000000
-Handler(tag: 111, JOBSTATUS) unregistering (send)
-2013-11-07 11:08:44,533-0600 INFO RequestHandler Handler(tag: 111, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 11:08:44,547-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50662] for addition
-2013-11-07 11:08:44,548-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50662] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:50662]
-2013-11-07 11:08:44,548-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:50662]
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f58
-2013-11-07 11:08:44,573-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f58
-MetaChannel [context: cps-50662, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50662]
-2013-11-07 11:08:44,574-0600 INFO MetaChannel MetaChannel [context: cps-50662, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50662]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-07 11:08:44,574-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-Avg stream buf: 6
-2013-11-07 11:08:44,594-0600 INFO Multiplexer Avg stream buf: 6
-Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-07 11:08:44,614-0600 INFO RequestHandler Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-07 11:08:44,695-0600 INFO RequestHandler Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-07 11:08:46,365-0600 DEBUG Cpu Completed
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:08:46,365-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 11:08:46,365-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-Channel config: null -> {keepalive=null}
-2013-11-07 11:08:46,365-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-07 11:08:46,365-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-TCPChannel [type: server, contact: 127.0.0.1:50660] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:08:46,365-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50660] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 7559, JOBSTATUS) unregistering (send)
-2013-11-07 11:08:46,365-0600 INFO RequestHandler Handler(tag: 7559, JOBSTATUS) unregistering (send)
-2013-11-07 11:08:46,365-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-null: Channel shut down
-2013-11-07 11:08:46,366-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-50660, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50660]] binding to NullChannel
-2013-11-07 11:08:46,366-0600 INFO MetaChannel MetaChannel [context: cps-50660, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50660]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50660]
-2013-11-07 11:08:46,366-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50660]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:08:46,366-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 11:08:46,366-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:08:46,366-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 11:08:46,366-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:08:46,366-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 11:08:46,366-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:08:46,366-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 11:08:46,366-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:08:46,366-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 11:08:46,367-0600 INFO ChannelManager Channel exception handled
-grind.mcs.anl.gov:0 (quality: 114.0) submitting urn:job-2-1383759274948-1383759274949: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 11:08:46,367-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 114.0) submitting urn:job-2-1383759274948-1383759274949: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:08:46,367-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel config: null -> {keepalive=null}
-2013-11-07 11:08:46,367-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50662] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:08:46,367-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50662] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-07 11:08:46,367-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-50662, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50662]] binding to NullChannel
-2013-11-07 11:08:46,367-0600 INFO MetaChannel MetaChannel [context: cps-50662, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50662]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50662]
-2013-11-07 11:08:46,367-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50662]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:08:46,367-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 11:08:46,368-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:08:46,368-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 11:08:46,368-0600 INFO ChannelManager Channel exception handled
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 11:08:46,371-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 11:08:46,371-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 11:08:46,372-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7560, JOBSTATUS) unregistering (send)
-2013-11-07 11:08:46,372-0600 INFO RequestHandler Handler(tag: 7560, JOBSTATUS) unregistering (send)
-Channel is closed. Removing.
-2013-11-07 11:08:46,380-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 11:08:46,380-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-07 11:08:46,380-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50660] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 11:08:46,380-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50660] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 11:08:46,380-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50660]
-2013-11-07 11:08:46,380-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50660]
-Channel is closed. Removing.
-2013-11-07 11:08:46,380-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 11:08:46,380-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-07 11:08:46,380-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50662] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 11:08:46,381-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50662] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 11:08:46,381-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50662]
-2013-11-07 11:08:46,381-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50662]
-2013-11-07 11:08:46,563-0600 DEBUG Cpu Completed
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:08:46,563-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-crush.mcs.anl.gov:0 jobTerminated
-2013-11-07 11:08:46,563-0600 INFO Cpu crush.mcs.anl.gov:0 jobTerminated
-Channel config: null -> {}
-2013-11-07 11:08:46,563-0600 INFO AbstractCoasterChannel Channel config: null -> {}
-2013-11-07 11:08:46,563-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-TCPChannel [type: server, contact: 127.0.0.1:50661] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:08:46,564-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50661] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 112, JOBSTATUS) unregistering (send)
-2013-11-07 11:08:46,564-0600 INFO RequestHandler Handler(tag: 112, JOBSTATUS) unregistering (send)
-2013-11-07 11:08:46,564-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-null: Channel shut down
-2013-11-07 11:08:46,564-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-50661, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50661]] binding to NullChannel
-2013-11-07 11:08:46,564-0600 INFO MetaChannel MetaChannel [context: cps-50661, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50661]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50661]
-2013-11-07 11:08:46,564-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50661]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:08:46,564-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 11:08:46,564-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:08:46,564-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 11:08:46,564-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:08:46,564-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 11:08:46,565-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:08:46,565-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 11:08:46,565-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:08:46,565-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 11:08:46,565-0600 INFO ChannelManager Channel exception handled
-2013-11-07 11:08:46,565-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-Channel is closed. Removing.
-2013-11-07 11:08:46,581-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 11:08:46,581-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {}
-2013-11-07 11:08:46,581-0600 INFO AbstractCoasterChannel Channel config: null -> {}
-TCPChannel [type: server, contact: 127.0.0.1:50661] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 11:08:46,581-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50661] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 11:08:46,581-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50661]
-2013-11-07 11:08:46,582-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50661]
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 11:08:48,384-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 11:08:48,384-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 11:08:48,384-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 11:08:48,384-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-07 11:08:48,385-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7561, JOBSTATUS) unregistering (send)
-2013-11-07 11:08:48,385-0600 INFO RequestHandler Handler(tag: 7561, JOBSTATUS) unregistering (send)
-2013-11-07 11:08:48,385-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-No streams
-2013-11-07 11:08:54,604-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:09:04,611-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:09:06,307-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:09:11,325-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:09:14,619-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:09:24,627-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:09:34,634-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:09:42,781-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:09:44,642-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:09:48,643-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:09:54,650-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:10:04,658-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:10:14,666-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:10:19,255-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:10:24,675-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 11:10:25,961-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 11:10:34,683-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:10:44,690-0600 INFO Multiplexer No streams
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110844.289, now=131107-111045.132, channel=TCPChannel [type: server, contact: 127.0.0.1:50660]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:10:45,132-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110844.289, now=131107-111045.132, channel=TCPChannel [type: server, contact: 127.0.0.1:50660]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 11:10:45,132-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110844.289, now=131107-111045.132, channel=TCPChannel [type: server, contact: 127.0.0.1:50660]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:10:45,132-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110844.289, now=131107-111045.132, channel=TCPChannel [type: server, contact: 127.0.0.1:50660]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 11:10:45,132-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110844.289, now=131107-111045.132, channel=TCPChannel [type: server, contact: 127.0.0.1:50660]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:10:45,132-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110844.289, now=131107-111045.132, channel=TCPChannel [type: server, contact: 127.0.0.1:50660]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 11:10:45,132-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110844.289, now=131107-111045.132, channel=TCPChannel [type: server, contact: 127.0.0.1:50660]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:10:45,132-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110844.289, now=131107-111045.132, channel=TCPChannel [type: server, contact: 127.0.0.1:50660]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 11:10:45,133-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110844.289, now=131107-111045.132, channel=TCPChannel [type: server, contact: 127.0.0.1:50660]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:10:45,133-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110844.289, now=131107-111045.132, channel=TCPChannel [type: server, contact: 127.0.0.1:50660]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-07 11:10:45,133-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50660] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110844.289, now=131107-111045.132, channel=TCPChannel [type: server, contact: 127.0.0.1:50660]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:10:45,133-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50660] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110844.289, now=131107-111045.132, channel=TCPChannel [type: server, contact: 127.0.0.1:50660]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 11:10:45,133-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110844.490, now=131107-111045.348, channel=TCPChannel [type: server, contact: 127.0.0.1:50661]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:10:45,348-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110844.490, now=131107-111045.348, channel=TCPChannel [type: server, contact: 127.0.0.1:50661]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 11:10:45,348-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110844.490, now=131107-111045.348, channel=TCPChannel [type: server, contact: 127.0.0.1:50661]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:10:45,348-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110844.490, now=131107-111045.348, channel=TCPChannel [type: server, contact: 127.0.0.1:50661]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 11:10:45,348-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110844.490, now=131107-111045.348, channel=TCPChannel [type: server, contact: 127.0.0.1:50661]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:10:45,348-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110844.490, now=131107-111045.348, channel=TCPChannel [type: server, contact: 127.0.0.1:50661]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 11:10:45,348-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110844.490, now=131107-111045.348, channel=TCPChannel [type: server, contact: 127.0.0.1:50661]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:10:45,348-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110844.490, now=131107-111045.348, channel=TCPChannel [type: server, contact: 127.0.0.1:50661]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 11:10:45,348-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110844.490, now=131107-111045.348, channel=TCPChannel [type: server, contact: 127.0.0.1:50661]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:10:45,348-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110844.490, now=131107-111045.348, channel=TCPChannel [type: server, contact: 127.0.0.1:50661]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {}
-2013-11-07 11:10:45,348-0600 INFO AbstractCoasterChannel Channel config: null -> {}
-TCPChannel [type: server, contact: 127.0.0.1:50661] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110844.490, now=131107-111045.348, channel=TCPChannel [type: server, contact: 127.0.0.1:50661]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:10:45,348-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50661] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110844.490, now=131107-111045.348, channel=TCPChannel [type: server, contact: 127.0.0.1:50661]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 11:10:45,348-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110844.694, now=131107-111045.548, channel=TCPChannel [type: server, contact: 127.0.0.1:50662]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:10:45,548-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110844.694, now=131107-111045.548, channel=TCPChannel [type: server, contact: 127.0.0.1:50662]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 11:10:45,548-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110844.694, now=131107-111045.548, channel=TCPChannel [type: server, contact: 127.0.0.1:50662]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:10:45,548-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110844.694, now=131107-111045.548, channel=TCPChannel [type: server, contact: 127.0.0.1:50662]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-07 11:10:45,548-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50662] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110844.694, now=131107-111045.548, channel=TCPChannel [type: server, contact: 127.0.0.1:50662]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:10:45,548-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50662] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-110844.694, now=131107-111045.548, channel=TCPChannel [type: server, contact: 127.0.0.1:50662]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 11:10:45,548-0600 INFO ChannelManager Channel already shut down
-No streams
-2013-11-07 11:10:54,698-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:10:55,729-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 11:11:03,279-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 11:11:04,706-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:11:14,714-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:11:24,722-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:11:32,203-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:11:34,730-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:11:40,597-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:11:44,738-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:11:54,746-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:12:04,754-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:12:08,677-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:12:14,763-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 11:12:17,916-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 11:12:24,771-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:12:34,779-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:12:44,787-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:12:45,151-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:12:54,796-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:12:55,233-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:13:04,804-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:13:14,812-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:13:21,625-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:13:24,820-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:13:32,551-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:13:34,828-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:13:44,836-0600 INFO Multiplexer No streams
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 11:13:53,227-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50668] for addition
-2013-11-07 11:13:53,228-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50668] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:50668]
-2013-11-07 11:13:53,228-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:50668]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f57
-2013-11-07 11:13:53,260-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f57
-MetaChannel [context: cps-50668, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50668]
-2013-11-07 11:13:53,260-0600 INFO MetaChannel MetaChannel [context: cps-50668, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50668]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-07 11:13:53,260-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-07 11:13:53,300-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-07 11:13:53,300-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-07 11:13:53,381-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-07 11:13:53,427-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 115.0) submitting urn:job-0-1383759274951-1383759274952: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 11:13:53,427-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 115.0) submitting urn:job-0-1383759274951-1383759274952: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 11:13:53,427-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-07 11:13:53,427-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 11:13:53,440-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50669] for addition
-2013-11-07 11:13:53,440-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50669] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:50669]
-2013-11-07 11:13:53,440-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:50669]
-2013-11-07 11:13:53,444-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7562, JOBSTATUS) unregistering (send)
-2013-11-07 11:13:53,444-0600 INFO RequestHandler Handler(tag: 7562, JOBSTATUS) unregistering (send)
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f56
-2013-11-07 11:13:53,462-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f56
-MetaChannel [context: cps-50669, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50669]
-2013-11-07 11:13:53,462-0600 INFO MetaChannel MetaChannel [context: cps-50669, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50669]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-07 11:13:53,462-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-07 11:13:53,502-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-07 11:13:53,583-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-07 11:13:53,628-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-crush.mcs.anl.gov:0 (quality: 41.0) submitting urn:job-1-1383759274954-1383759274955: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 11:13:53,628-0600 INFO Cpu crush.mcs.anl.gov:0 (quality: 41.0) submitting urn:job-1-1383759274954-1383759274955: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 11:13:53,633-0600 DEBUG Cpu Active workerid=crush.mcs.anl.gov:000000
-Handler(tag: 113, JOBSTATUS) unregistering (send)
-2013-11-07 11:13:53,633-0600 INFO RequestHandler Handler(tag: 113, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 11:13:53,639-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50670] for addition
-2013-11-07 11:13:53,640-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50670] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:50670]
-2013-11-07 11:13:53,640-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:50670]
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f55
-2013-11-07 11:13:53,664-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f55
-MetaChannel [context: cps-50670, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50670]
-2013-11-07 11:13:53,664-0600 INFO MetaChannel MetaChannel [context: cps-50670, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50670]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-07 11:13:53,664-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-07 11:13:53,704-0600 INFO RequestHandler Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-07 11:13:53,785-0600 INFO RequestHandler Handler(tag: 42566, SUBMITJOB) unregistering (send)
-Avg stream buf: 0
-2013-11-07 11:13:54,848-0600 INFO Multiplexer Avg stream buf: 0
-2013-11-07 11:13:55,497-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 11:13:55,497-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:13:55,497-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:13:55,497-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Channel config: null -> {keepalive=null}
-2013-11-07 11:13:55,497-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-07 11:13:55,497-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-Handler(tag: 7563, JOBSTATUS) unregistering (send)
-2013-11-07 11:13:55,498-0600 INFO RequestHandler Handler(tag: 7563, JOBSTATUS) unregistering (send)
-grind.mcs.anl.gov:0 (quality: 116.0) submitting urn:job-2-1383759274957-1383759274958: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 11:13:55,498-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 116.0) submitting urn:job-2-1383759274957-1383759274958: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-TCPChannel [type: server, contact: 127.0.0.1:50668] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:13:55,498-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50668] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-07 11:13:55,499-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-50668, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50668]] binding to NullChannel
-2013-11-07 11:13:55,499-0600 INFO MetaChannel MetaChannel [context: cps-50668, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50668]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50668]
-2013-11-07 11:13:55,499-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50668]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:13:55,499-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 11:13:55,499-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:13:55,499-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 11:13:55,499-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:13:55,499-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 11:13:55,499-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:13:55,499-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 11:13:55,499-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:13:55,499-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 11:13:55,500-0600 INFO ChannelManager Channel exception handled
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:13:55,500-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel config: null -> {keepalive=null}
-2013-11-07 11:13:55,500-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50670] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:13:55,500-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50670] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-07 11:13:55,500-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-50670, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50670]] binding to NullChannel
-2013-11-07 11:13:55,500-0600 INFO MetaChannel MetaChannel [context: cps-50670, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50670]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50670]
-2013-11-07 11:13:55,500-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50670]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:13:55,500-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 11:13:55,500-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:13:55,500-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 11:13:55,500-0600 INFO ChannelManager Channel exception handled
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 11:13:55,503-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 11:13:55,503-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 11:13:55,503-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7564, JOBSTATUS) unregistering (send)
-2013-11-07 11:13:55,503-0600 INFO RequestHandler Handler(tag: 7564, JOBSTATUS) unregistering (send)
-Channel is closed. Removing.
-2013-11-07 11:13:55,510-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 11:13:55,510-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-07 11:13:55,510-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50668] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 11:13:55,510-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50668] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 11:13:55,510-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50668]
-2013-11-07 11:13:55,510-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50668]
-Channel is closed. Removing.
-2013-11-07 11:13:55,510-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 11:13:55,510-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-07 11:13:55,510-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50670] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 11:13:55,510-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50670] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 11:13:55,510-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50670]
-2013-11-07 11:13:55,511-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50670]
-2013-11-07 11:13:55,892-0600 DEBUG Cpu Completed
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:13:55,892-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-crush.mcs.anl.gov:0 jobTerminated
-2013-11-07 11:13:55,892-0600 INFO Cpu crush.mcs.anl.gov:0 jobTerminated
-Channel config: null -> {}
-2013-11-07 11:13:55,892-0600 INFO AbstractCoasterChannel Channel config: null -> {}
-2013-11-07 11:13:55,892-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-TCPChannel [type: server, contact: 127.0.0.1:50669] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:13:55,893-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50669] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 114, JOBSTATUS) unregistering (send)
-2013-11-07 11:13:55,893-0600 INFO RequestHandler Handler(tag: 114, JOBSTATUS) unregistering (send)
-2013-11-07 11:13:55,893-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-07 11:13:55,893-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-null: Channel shut down
-2013-11-07 11:13:55,893-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-50669, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50669]] binding to NullChannel
-2013-11-07 11:13:55,893-0600 INFO MetaChannel MetaChannel [context: cps-50669, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50669]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50669]
-2013-11-07 11:13:55,893-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50669]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:13:55,893-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 11:13:55,893-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:13:55,893-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 11:13:55,893-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:13:55,893-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 11:13:55,893-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:13:55,893-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 11:13:55,894-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:13:55,894-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 11:13:55,894-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-07 11:13:55,912-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 11:13:55,912-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {}
-2013-11-07 11:13:55,912-0600 INFO AbstractCoasterChannel Channel config: null -> {}
-TCPChannel [type: server, contact: 127.0.0.1:50669] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 11:13:55,912-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50669] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 11:13:55,912-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50669]
-2013-11-07 11:13:55,912-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50669]
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 11:13:57,569-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 11:13:57,569-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 11:13:57,569-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 11:13:57,569-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-07 11:13:57,569-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7565, JOBSTATUS) unregistering (send)
-2013-11-07 11:13:57,569-0600 INFO RequestHandler Handler(tag: 7565, JOBSTATUS) unregistering (send)
-2013-11-07 11:13:57,569-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 11:13:58,099-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 11:14:04,857-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:14:09,869-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:14:14,865-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:14:24,874-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:14:34,573-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:14:34,882-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:14:44,890-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:14:47,187-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:14:54,898-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:15:04,906-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:15:11,047-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:15:14,914-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 11:15:24,505-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 11:15:24,922-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:15:34,930-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:15:44,938-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:15:47,521-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-111353.381, now=131107-111554.228, channel=TCPChannel [type: server, contact: 127.0.0.1:50668]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:15:54,228-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-111353.381, now=131107-111554.228, channel=TCPChannel [type: server, contact: 127.0.0.1:50668]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 11:15:54,228-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-111353.381, now=131107-111554.228, channel=TCPChannel [type: server, contact: 127.0.0.1:50668]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:15:54,228-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-111353.381, now=131107-111554.228, channel=TCPChannel [type: server, contact: 127.0.0.1:50668]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 11:15:54,228-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-111353.381, now=131107-111554.228, channel=TCPChannel [type: server, contact: 127.0.0.1:50668]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:15:54,228-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-111353.381, now=131107-111554.228, channel=TCPChannel [type: server, contact: 127.0.0.1:50668]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 11:15:54,228-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-111353.381, now=131107-111554.228, channel=TCPChannel [type: server, contact: 127.0.0.1:50668]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:15:54,228-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-111353.381, now=131107-111554.228, channel=TCPChannel [type: server, contact: 127.0.0.1:50668]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 11:15:54,229-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-111353.381, now=131107-111554.228, channel=TCPChannel [type: server, contact: 127.0.0.1:50668]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:15:54,229-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-111353.381, now=131107-111554.228, channel=TCPChannel [type: server, contact: 127.0.0.1:50668]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-07 11:15:54,229-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50668] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-111353.381, now=131107-111554.228, channel=TCPChannel [type: server, contact: 127.0.0.1:50668]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:15:54,229-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50668] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-111353.381, now=131107-111554.228, channel=TCPChannel [type: server, contact: 127.0.0.1:50668]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 11:15:54,229-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-111353.582, now=131107-111554.440, channel=TCPChannel [type: server, contact: 127.0.0.1:50669]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:15:54,440-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-111353.582, now=131107-111554.440, channel=TCPChannel [type: server, contact: 127.0.0.1:50669]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 11:15:54,440-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-111353.582, now=131107-111554.440, channel=TCPChannel [type: server, contact: 127.0.0.1:50669]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:15:54,440-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-111353.582, now=131107-111554.440, channel=TCPChannel [type: server, contact: 127.0.0.1:50669]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 11:15:54,440-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-111353.582, now=131107-111554.440, channel=TCPChannel [type: server, contact: 127.0.0.1:50669]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:15:54,440-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-111353.582, now=131107-111554.440, channel=TCPChannel [type: server, contact: 127.0.0.1:50669]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 11:15:54,440-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-111353.582, now=131107-111554.440, channel=TCPChannel [type: server, contact: 127.0.0.1:50669]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:15:54,440-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-111353.582, now=131107-111554.440, channel=TCPChannel [type: server, contact: 127.0.0.1:50669]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 11:15:54,440-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-111353.582, now=131107-111554.440, channel=TCPChannel [type: server, contact: 127.0.0.1:50669]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:15:54,440-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-111353.582, now=131107-111554.440, channel=TCPChannel [type: server, contact: 127.0.0.1:50669]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {}
-2013-11-07 11:15:54,440-0600 INFO AbstractCoasterChannel Channel config: null -> {}
-TCPChannel [type: server, contact: 127.0.0.1:50669] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-111353.582, now=131107-111554.440, channel=TCPChannel [type: server, contact: 127.0.0.1:50669]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:15:54,440-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50669] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-111353.582, now=131107-111554.440, channel=TCPChannel [type: server, contact: 127.0.0.1:50669]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 11:15:54,440-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-111353.785, now=131107-111554.639, channel=TCPChannel [type: server, contact: 127.0.0.1:50670]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:15:54,639-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-111353.785, now=131107-111554.639, channel=TCPChannel [type: server, contact: 127.0.0.1:50670]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 11:15:54,640-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-111353.785, now=131107-111554.639, channel=TCPChannel [type: server, contact: 127.0.0.1:50670]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:15:54,640-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-111353.785, now=131107-111554.639, channel=TCPChannel [type: server, contact: 127.0.0.1:50670]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-07 11:15:54,640-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50670] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-111353.785, now=131107-111554.639, channel=TCPChannel [type: server, contact: 127.0.0.1:50670]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:15:54,640-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50670] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-111353.785, now=131107-111554.639, channel=TCPChannel [type: server, contact: 127.0.0.1:50670]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 11:15:54,640-0600 INFO ChannelManager Channel already shut down
-No streams
-2013-11-07 11:15:54,946-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:16:01,823-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:16:04,954-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:16:14,962-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 11:16:23,995-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 11:16:24,970-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:16:34,978-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 11:16:39,141-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 11:16:44,986-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:16:54,994-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:17:00,469-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:17:05,002-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:17:15,010-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:17:16,459-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:17:25,018-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:17:35,026-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:17:36,943-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:17:45,034-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 11:17:53,777-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 11:17:55,042-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:18:05,050-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:18:13,417-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:18:15,058-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:18:25,067-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:18:31,095-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:18:35,074-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:18:45,083-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 11:18:49,891-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 11:18:55,091-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:19:05,099-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:19:08,414-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:19:15,107-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:19:25,115-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:19:26,365-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:19:35,123-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:19:45,131-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:19:45,732-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:19:55,139-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:20:02,839-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:20:05,147-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:20:15,155-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:20:23,050-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:20:25,163-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:20:35,171-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:20:39,313-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:20:45,179-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:20:55,187-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 11:21:00,368-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 11:21:05,195-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:21:15,203-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:21:15,787-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:21:25,211-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:21:35,219-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:21:37,686-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:21:45,227-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:21:52,261-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:21:55,235-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:22:05,243-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 11:22:15,004-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 11:22:15,251-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:22:25,259-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:22:28,735-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 11:22:31,676-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50671] for addition
-2013-11-07 11:22:31,676-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50671] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:50671]
-2013-11-07 11:22:31,676-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:50671]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f54
-2013-11-07 11:22:31,698-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f54
-MetaChannel [context: cps-50671, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50671]
-2013-11-07 11:22:31,698-0600 INFO MetaChannel MetaChannel [context: cps-50671, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50671]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-07 11:22:31,698-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-07 11:22:31,758-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-07 11:22:31,758-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-07 11:22:31,839-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-07 11:22:31,849-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 117.0) submitting urn:job-0-1383759274960-1383759274961: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 11:22:31,849-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 117.0) submitting urn:job-0-1383759274960-1383759274961: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 11:22:31,850-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-07 11:22:31,850-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-07 11:22:31,855-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7566, JOBSTATUS) unregistering (send)
-2013-11-07 11:22:31,855-0600 INFO RequestHandler Handler(tag: 7566, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 11:22:31,900-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50672] for addition
-2013-11-07 11:22:31,900-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50672] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:50672]
-2013-11-07 11:22:31,900-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:50672]
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f53
-2013-11-07 11:22:31,940-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f53
-MetaChannel [context: cps-50672, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50672]
-2013-11-07 11:22:31,940-0600 INFO MetaChannel MetaChannel [context: cps-50672, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50672]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-07 11:22:31,940-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-07 11:22:31,961-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-07 11:22:32,041-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-07 11:22:32,050-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-crush.mcs.anl.gov:0 (quality: 42.0) submitting urn:job-1-1383759274963-1383759274964: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 11:22:32,050-0600 INFO Cpu crush.mcs.anl.gov:0 (quality: 42.0) submitting urn:job-1-1383759274963-1383759274964: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 11:22:32,057-0600 DEBUG Cpu Active workerid=crush.mcs.anl.gov:000000
-Handler(tag: 115, JOBSTATUS) unregistering (send)
-2013-11-07 11:22:32,057-0600 INFO RequestHandler Handler(tag: 115, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 11:22:32,102-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50673] for addition
-2013-11-07 11:22:32,102-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50673] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:50673]
-2013-11-07 11:22:32,102-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:50673]
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f52
-2013-11-07 11:22:32,142-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f52
-MetaChannel [context: cps-50673, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50673]
-2013-11-07 11:22:32,142-0600 INFO MetaChannel MetaChannel [context: cps-50673, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50673]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-07 11:22:32,142-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-07 11:22:32,202-0600 INFO RequestHandler Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-07 11:22:32,283-0600 INFO RequestHandler Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-07 11:22:34,099-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 11:22:34,099-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 11:22:34,099-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:22:34,099-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 7567, JOBSTATUS) unregistering (send)
-2013-11-07 11:22:34,099-0600 INFO RequestHandler Handler(tag: 7567, JOBSTATUS) unregistering (send)
-2013-11-07 11:22:34,099-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-Channel config: null -> {keepalive=null}
-2013-11-07 11:22:34,099-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50671] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:22:34,099-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50671] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-grind.mcs.anl.gov:0 (quality: 118.0) submitting urn:job-2-1383759274966-1383759274967: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 11:22:34,099-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 118.0) submitting urn:job-2-1383759274966-1383759274967: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-null: Channel shut down
-2013-11-07 11:22:34,099-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-50671, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50671]] binding to NullChannel
-2013-11-07 11:22:34,099-0600 INFO MetaChannel MetaChannel [context: cps-50671, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50671]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50671]
-2013-11-07 11:22:34,100-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50671]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:22:34,100-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 11:22:34,100-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:22:34,100-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 11:22:34,100-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:22:34,100-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 11:22:34,100-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:22:34,100-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 11:22:34,100-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:22:34,100-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 11:22:34,100-0600 INFO ChannelManager Channel exception handled
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:22:34,100-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel config: null -> {keepalive=null}
-2013-11-07 11:22:34,100-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50673] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:22:34,100-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50673] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-07 11:22:34,101-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-50673, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50673]] binding to NullChannel
-2013-11-07 11:22:34,101-0600 INFO MetaChannel MetaChannel [context: cps-50673, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50673]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50673]
-2013-11-07 11:22:34,101-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50673]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:22:34,101-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 11:22:34,101-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:22:34,101-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 11:22:34,101-0600 INFO ChannelManager Channel exception handled
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 11:22:34,104-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 11:22:34,105-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 11:22:34,105-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7568, JOBSTATUS) unregistering (send)
-2013-11-07 11:22:34,105-0600 INFO RequestHandler Handler(tag: 7568, JOBSTATUS) unregistering (send)
-Channel is closed. Removing.
-2013-11-07 11:22:34,108-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 11:22:34,108-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-07 11:22:34,108-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50671] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 11:22:34,108-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50671] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 11:22:34,108-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50671]
-2013-11-07 11:22:34,108-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50671]
-Channel is closed. Removing.
-2013-11-07 11:22:34,108-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 11:22:34,109-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-07 11:22:34,109-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50673] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 11:22:34,109-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50673] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 11:22:34,109-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50673]
-2013-11-07 11:22:34,109-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50673]
-2013-11-07 11:22:34,513-0600 DEBUG Cpu Completed
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:22:34,513-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-crush.mcs.anl.gov:0 jobTerminated
-2013-11-07 11:22:34,513-0600 INFO Cpu crush.mcs.anl.gov:0 jobTerminated
-Channel config: null -> {keepalive=null}
-2013-11-07 11:22:34,513-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-07 11:22:34,513-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-TCPChannel [type: server, contact: 127.0.0.1:50672] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:22:34,513-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50672] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 116, JOBSTATUS) unregistering (send)
-2013-11-07 11:22:34,513-0600 INFO RequestHandler Handler(tag: 116, JOBSTATUS) unregistering (send)
-2013-11-07 11:22:34,513-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-null: Channel shut down
-2013-11-07 11:22:34,513-0600 INFO AbstractTCPChannel null: Channel shut down
-2013-11-07 11:22:34,513-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-MetaChannel [context: cps-50672, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50672]] binding to NullChannel
-2013-11-07 11:22:34,514-0600 INFO MetaChannel MetaChannel [context: cps-50672, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50672]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50672]
-2013-11-07 11:22:34,514-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50672]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:22:34,514-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 11:22:34,514-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:22:34,514-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 11:22:34,514-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:22:34,514-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 11:22:34,514-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:22:34,514-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 11:22:34,514-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:22:34,514-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 11:22:34,514-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-07 11:22:34,530-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 11:22:34,530-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-07 11:22:34,530-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50672] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 11:22:34,530-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50672] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 11:22:34,530-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50672]
-2013-11-07 11:22:34,530-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50672]
-No streams
-2013-11-07 11:22:35,272-0600 INFO Multiplexer No streams
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 11:22:36,118-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 11:22:36,118-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 11:22:36,118-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 11:22:36,118-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-07 11:22:36,118-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7569, JOBSTATUS) unregistering (send)
-2013-11-07 11:22:36,118-0600 INFO RequestHandler Handler(tag: 7569, JOBSTATUS) unregistering (send)
-2013-11-07 11:22:36,118-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-No streams
-2013-11-07 11:22:45,280-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:22:52,322-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:22:55,288-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:23:05,209-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:23:05,297-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:23:15,305-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:23:25,313-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:23:29,641-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:23:35,321-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:23:41,683-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:23:45,329-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:23:55,337-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:24:05,345-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:24:06,959-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 11:24:12,869-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50674] for addition
-2013-11-07 11:24:12,869-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50674] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:50674]
-2013-11-07 11:24:12,869-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:50674]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f51
-2013-11-07 11:24:12,907-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f51
-MetaChannel [context: cps-50674, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50674]
-2013-11-07 11:24:12,907-0600 INFO MetaChannel MetaChannel [context: cps-50674, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50674]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-07 11:24:12,907-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-07 11:24:12,928-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-07 11:24:12,928-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-07 11:24:13,008-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-07 11:24:13,028-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 119.0) submitting urn:job-0-1383759274969-1383759274970: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 11:24:13,029-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 119.0) submitting urn:job-0-1383759274969-1383759274970: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 11:24:13,029-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-07 11:24:13,029-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 11:24:13,068-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50675] for addition
-2013-11-07 11:24:13,068-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50675] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:50675]
-2013-11-07 11:24:13,069-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:50675]
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f50
-2013-11-07 11:24:13,089-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f50
-MetaChannel [context: cps-50675, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50675]
-2013-11-07 11:24:13,089-0600 INFO MetaChannel MetaChannel [context: cps-50675, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50675]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-07 11:24:13,089-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-07 11:24:13,094-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7570, JOBSTATUS) unregistering (send)
-2013-11-07 11:24:13,094-0600 INFO RequestHandler Handler(tag: 7570, JOBSTATUS) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-07 11:24:13,130-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-07 11:24:13,210-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-07 11:24:13,229-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-crush.mcs.anl.gov:0 (quality: 43.0) submitting urn:job-1-1383759274972-1383759274973: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 11:24:13,230-0600 INFO Cpu crush.mcs.anl.gov:0 (quality: 43.0) submitting urn:job-1-1383759274972-1383759274973: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 11:24:13,236-0600 DEBUG Cpu Active workerid=crush.mcs.anl.gov:000000
-Handler(tag: 117, JOBSTATUS) unregistering (send)
-2013-11-07 11:24:13,236-0600 INFO RequestHandler Handler(tag: 117, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 11:24:13,268-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50676] for addition
-2013-11-07 11:24:13,268-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50676] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:50676]
-2013-11-07 11:24:13,268-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:50676]
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f4f
-2013-11-07 11:24:13,291-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f4f
-MetaChannel [context: cps-50676, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50676]
-2013-11-07 11:24:13,291-0600 INFO MetaChannel MetaChannel [context: cps-50676, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50676]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-07 11:24:13,291-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-07 11:24:13,332-0600 INFO RequestHandler Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-07 11:24:13,412-0600 INFO RequestHandler Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-07 11:24:15,214-0600 DEBUG Cpu Completed
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:24:15,215-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 11:24:15,215-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-Channel config: null -> {keepalive=null}
-2013-11-07 11:24:15,215-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-07 11:24:15,215-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-TCPChannel [type: server, contact: 127.0.0.1:50674] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:24:15,215-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50674] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 7571, JOBSTATUS) unregistering (send)
-2013-11-07 11:24:15,215-0600 INFO RequestHandler Handler(tag: 7571, JOBSTATUS) unregistering (send)
-2013-11-07 11:24:15,215-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-null: Channel shut down
-2013-11-07 11:24:15,215-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-50674, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50674]] binding to NullChannel
-2013-11-07 11:24:15,215-0600 INFO MetaChannel MetaChannel [context: cps-50674, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50674]] binding to NullChannel
-grind.mcs.anl.gov:0 (quality: 120.0) submitting urn:job-2-1383759274975-1383759274976: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 11:24:15,215-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 120.0) submitting urn:job-2-1383759274975-1383759274976: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50674]
-2013-11-07 11:24:15,215-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50674]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:24:15,215-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 11:24:15,216-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:24:15,216-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 11:24:15,216-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:24:15,216-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 11:24:15,216-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:24:15,216-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 11:24:15,216-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:24:15,216-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 11:24:15,216-0600 INFO ChannelManager Channel exception handled
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:24:15,216-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel config: null -> {keepalive=null}
-2013-11-07 11:24:15,216-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50676] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:24:15,216-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50676] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-07 11:24:15,217-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-50676, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50676]] binding to NullChannel
-2013-11-07 11:24:15,217-0600 INFO MetaChannel MetaChannel [context: cps-50676, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50676]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50676]
-2013-11-07 11:24:15,217-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50676]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:24:15,217-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 11:24:15,217-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:24:15,217-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 11:24:15,217-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-07 11:24:15,219-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 11:24:15,219-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-07 11:24:15,219-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50676] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 11:24:15,219-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50676] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 11:24:15,219-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50676]
-2013-11-07 11:24:15,219-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50676]
-Channel is closed. Removing.
-2013-11-07 11:24:15,219-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 11:24:15,219-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-07 11:24:15,219-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50674] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 11:24:15,219-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50674] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 11:24:15,219-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50674]
-2013-11-07 11:24:15,220-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50674]
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 11:24:15,221-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 11:24:15,221-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 11:24:15,221-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7572, JOBSTATUS) unregistering (send)
-2013-11-07 11:24:15,221-0600 INFO RequestHandler Handler(tag: 7572, JOBSTATUS) unregistering (send)
-2013-11-07 11:24:15,261-0600 DEBUG Cpu Completed
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:24:15,261-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-crush.mcs.anl.gov:0 jobTerminated
-2013-11-07 11:24:15,261-0600 INFO Cpu crush.mcs.anl.gov:0 jobTerminated
-Channel config: null -> {}
-2013-11-07 11:24:15,261-0600 INFO AbstractCoasterChannel Channel config: null -> {}
-2013-11-07 11:24:15,261-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-TCPChannel [type: server, contact: 127.0.0.1:50675] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:24:15,261-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50675] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 118, JOBSTATUS) unregistering (send)
-2013-11-07 11:24:15,261-0600 INFO RequestHandler Handler(tag: 118, JOBSTATUS) unregistering (send)
-2013-11-07 11:24:15,261-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-null: Channel shut down
-2013-11-07 11:24:15,261-0600 INFO AbstractTCPChannel null: Channel shut down
-2013-11-07 11:24:15,261-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-MetaChannel [context: cps-50675, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50675]] binding to NullChannel
-2013-11-07 11:24:15,261-0600 INFO MetaChannel MetaChannel [context: cps-50675, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50675]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50675]
-2013-11-07 11:24:15,261-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50675]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:24:15,261-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 11:24:15,262-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:24:15,262-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 11:24:15,262-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:24:15,262-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 11:24:15,262-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:24:15,262-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 11:24:15,262-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 11:24:15,262-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 11:24:15,262-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-07 11:24:15,280-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 11:24:15,280-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {}
-2013-11-07 11:24:15,280-0600 INFO AbstractCoasterChannel Channel config: null -> {}
-TCPChannel [type: server, contact: 127.0.0.1:50675] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 11:24:15,280-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50675] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 11:24:15,280-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50675]
-2013-11-07 11:24:15,280-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50675]
-No streams
-2013-11-07 11:24:15,360-0600 INFO Multiplexer No streams
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 11:24:17,236-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 11:24:17,236-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 11:24:17,236-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 11:24:17,236-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-07 11:24:17,236-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7573, JOBSTATUS) unregistering (send)
-2013-11-07 11:24:17,236-0600 INFO RequestHandler Handler(tag: 7573, JOBSTATUS) unregistering (send)
-2013-11-07 11:24:17,236-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 11:24:18,157-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 11:24:25,368-0600 INFO Multiplexer No streams
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112231.839, now=131107-112432.676, channel=TCPChannel [type: server, contact: 127.0.0.1:50671]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:24:32,676-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112231.839, now=131107-112432.676, channel=TCPChannel [type: server, contact: 127.0.0.1:50671]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 11:24:32,676-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112231.839, now=131107-112432.676, channel=TCPChannel [type: server, contact: 127.0.0.1:50671]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:24:32,676-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112231.839, now=131107-112432.676, channel=TCPChannel [type: server, contact: 127.0.0.1:50671]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 11:24:32,676-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112231.839, now=131107-112432.676, channel=TCPChannel [type: server, contact: 127.0.0.1:50671]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:24:32,676-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112231.839, now=131107-112432.676, channel=TCPChannel [type: server, contact: 127.0.0.1:50671]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 11:24:32,676-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112231.839, now=131107-112432.676, channel=TCPChannel [type: server, contact: 127.0.0.1:50671]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:24:32,676-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112231.839, now=131107-112432.676, channel=TCPChannel [type: server, contact: 127.0.0.1:50671]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 11:24:32,676-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112231.839, now=131107-112432.676, channel=TCPChannel [type: server, contact: 127.0.0.1:50671]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:24:32,676-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112231.839, now=131107-112432.676, channel=TCPChannel [type: server, contact: 127.0.0.1:50671]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-07 11:24:32,676-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50671] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112231.839, now=131107-112432.676, channel=TCPChannel [type: server, contact: 127.0.0.1:50671]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:24:32,676-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50671] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112231.839, now=131107-112432.676, channel=TCPChannel [type: server, contact: 127.0.0.1:50671]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 11:24:32,676-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112232.041, now=131107-112432.900, channel=TCPChannel [type: server, contact: 127.0.0.1:50672]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:24:32,900-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112232.041, now=131107-112432.900, channel=TCPChannel [type: server, contact: 127.0.0.1:50672]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 11:24:32,900-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112232.041, now=131107-112432.900, channel=TCPChannel [type: server, contact: 127.0.0.1:50672]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:24:32,900-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112232.041, now=131107-112432.900, channel=TCPChannel [type: server, contact: 127.0.0.1:50672]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 11:24:32,900-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112232.041, now=131107-112432.900, channel=TCPChannel [type: server, contact: 127.0.0.1:50672]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:24:32,900-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112232.041, now=131107-112432.900, channel=TCPChannel [type: server, contact: 127.0.0.1:50672]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 11:24:32,900-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112232.041, now=131107-112432.900, channel=TCPChannel [type: server, contact: 127.0.0.1:50672]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:24:32,900-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112232.041, now=131107-112432.900, channel=TCPChannel [type: server, contact: 127.0.0.1:50672]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 11:24:32,900-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112232.041, now=131107-112432.900, channel=TCPChannel [type: server, contact: 127.0.0.1:50672]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:24:32,900-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112232.041, now=131107-112432.900, channel=TCPChannel [type: server, contact: 127.0.0.1:50672]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-07 11:24:32,900-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50672] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112232.041, now=131107-112432.900, channel=TCPChannel [type: server, contact: 127.0.0.1:50672]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:24:32,900-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50672] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112232.041, now=131107-112432.900, channel=TCPChannel [type: server, contact: 127.0.0.1:50672]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 11:24:32,900-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112232.283, now=131107-112433.102, channel=TCPChannel [type: server, contact: 127.0.0.1:50673]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:24:33,102-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112232.283, now=131107-112433.102, channel=TCPChannel [type: server, contact: 127.0.0.1:50673]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 11:24:33,102-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112232.283, now=131107-112433.102, channel=TCPChannel [type: server, contact: 127.0.0.1:50673]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:24:33,102-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112232.283, now=131107-112433.102, channel=TCPChannel [type: server, contact: 127.0.0.1:50673]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-07 11:24:33,103-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50673] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112232.283, now=131107-112433.102, channel=TCPChannel [type: server, contact: 127.0.0.1:50673]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:24:33,103-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50673] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112232.283, now=131107-112433.102, channel=TCPChannel [type: server, contact: 127.0.0.1:50673]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 11:24:33,103-0600 INFO ChannelManager Channel already shut down
-No streams
-2013-11-07 11:24:35,376-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:24:44,277-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:24:45,383-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:24:54,631-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:24:55,391-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:25:05,399-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:25:15,407-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:25:21,595-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:25:25,414-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:25:31,105-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:25:35,422-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:25:45,430-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:25:55,438-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:25:58,913-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:26:05,446-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 11:26:07,580-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112413.008, now=131107-112613.869, channel=TCPChannel [type: server, contact: 127.0.0.1:50674]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:26:13,870-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112413.008, now=131107-112613.869, channel=TCPChannel [type: server, contact: 127.0.0.1:50674]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 11:26:13,870-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112413.008, now=131107-112613.869, channel=TCPChannel [type: server, contact: 127.0.0.1:50674]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:26:13,870-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112413.008, now=131107-112613.869, channel=TCPChannel [type: server, contact: 127.0.0.1:50674]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 11:26:13,870-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112413.008, now=131107-112613.869, channel=TCPChannel [type: server, contact: 127.0.0.1:50674]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:26:13,870-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112413.008, now=131107-112613.869, channel=TCPChannel [type: server, contact: 127.0.0.1:50674]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 11:26:13,870-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112413.008, now=131107-112613.869, channel=TCPChannel [type: server, contact: 127.0.0.1:50674]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:26:13,870-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112413.008, now=131107-112613.869, channel=TCPChannel [type: server, contact: 127.0.0.1:50674]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 11:26:13,870-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112413.008, now=131107-112613.869, channel=TCPChannel [type: server, contact: 127.0.0.1:50674]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:26:13,870-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112413.008, now=131107-112613.869, channel=TCPChannel [type: server, contact: 127.0.0.1:50674]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-07 11:26:13,870-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50674] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112413.008, now=131107-112613.869, channel=TCPChannel [type: server, contact: 127.0.0.1:50674]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:26:13,870-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50674] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112413.008, now=131107-112613.869, channel=TCPChannel [type: server, contact: 127.0.0.1:50674]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 11:26:13,870-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112413.210, now=131107-112614.068, channel=TCPChannel [type: server, contact: 127.0.0.1:50675]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:26:14,068-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112413.210, now=131107-112614.068, channel=TCPChannel [type: server, contact: 127.0.0.1:50675]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 11:26:14,068-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112413.210, now=131107-112614.068, channel=TCPChannel [type: server, contact: 127.0.0.1:50675]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:26:14,068-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112413.210, now=131107-112614.068, channel=TCPChannel [type: server, contact: 127.0.0.1:50675]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 11:26:14,068-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112413.210, now=131107-112614.068, channel=TCPChannel [type: server, contact: 127.0.0.1:50675]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:26:14,069-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112413.210, now=131107-112614.068, channel=TCPChannel [type: server, contact: 127.0.0.1:50675]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 11:26:14,069-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112413.210, now=131107-112614.068, channel=TCPChannel [type: server, contact: 127.0.0.1:50675]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:26:14,069-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112413.210, now=131107-112614.068, channel=TCPChannel [type: server, contact: 127.0.0.1:50675]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 11:26:14,069-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112413.210, now=131107-112614.068, channel=TCPChannel [type: server, contact: 127.0.0.1:50675]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:26:14,069-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112413.210, now=131107-112614.068, channel=TCPChannel [type: server, contact: 127.0.0.1:50675]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {}
-2013-11-07 11:26:14,069-0600 INFO AbstractCoasterChannel Channel config: null -> {}
-TCPChannel [type: server, contact: 127.0.0.1:50675] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112413.210, now=131107-112614.068, channel=TCPChannel [type: server, contact: 127.0.0.1:50675]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:26:14,069-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50675] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112413.210, now=131107-112614.068, channel=TCPChannel [type: server, contact: 127.0.0.1:50675]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 11:26:14,069-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112413.412, now=131107-112614.268, channel=TCPChannel [type: server, contact: 127.0.0.1:50676]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:26:14,268-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112413.412, now=131107-112614.268, channel=TCPChannel [type: server, contact: 127.0.0.1:50676]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 11:26:14,268-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112413.412, now=131107-112614.268, channel=TCPChannel [type: server, contact: 127.0.0.1:50676]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:26:14,268-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112413.412, now=131107-112614.268, channel=TCPChannel [type: server, contact: 127.0.0.1:50676]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-07 11:26:14,268-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50676] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112413.412, now=131107-112614.268, channel=TCPChannel [type: server, contact: 127.0.0.1:50676]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 11:26:14,268-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50676] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-112413.412, now=131107-112614.268, channel=TCPChannel [type: server, contact: 127.0.0.1:50676]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 11:26:14,268-0600 INFO ChannelManager Channel already shut down
-No streams
-2013-11-07 11:26:15,453-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:26:25,461-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:26:35,470-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 11:26:36,231-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:26:44,053-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:26:45,478-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:26:55,486-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:27:05,494-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-07 11:27:13,548-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-07 11:27:15,502-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-07 11:27:20,527-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-07 11:27:25,510-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:27:35,518-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:27:45,526-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 11:27:50,867-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 11:27:55,534-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:27:57,002-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:28:05,543-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:28:15,551-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:28:25,559-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:28:28,185-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-07 11:28:33,476-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-07 11:28:35,567-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:28:45,576-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:28:55,584-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 11:29:05,503-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 11:29:05,592-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 11:29:09,950-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 11:29:15,600-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:29:25,609-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:29:35,617-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:29:42,821-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:29:45,625-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 11:29:46,424-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 11:29:55,633-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:30:05,642-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:30:15,650-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:30:20,140-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 11:30:22,898-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 11:30:25,658-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:30:35,666-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:30:45,675-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:30:55,683-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 11:30:57,458-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:30:59,373-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:31:05,691-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:31:15,699-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:31:25,708-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:31:34,776-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:31:35,716-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:31:35,847-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:31:45,724-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:31:55,732-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:32:05,740-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 11:32:12,095-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:32:12,321-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:32:15,749-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:32:25,757-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:32:35,765-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:32:45,774-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:32:48,795-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 11:32:49,412-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 11:32:55,782-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:33:05,790-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:33:15,798-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:33:25,270-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:33:25,806-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:33:26,730-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:33:35,815-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:33:45,823-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:33:55,831-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:34:01,743-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 11:34:04,048-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 11:34:05,839-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:34:15,847-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:34:25,855-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:34:35,863-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 11:34:38,218-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:34:41,366-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:34:45,871-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:34:55,879-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:35:05,888-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:35:14,691-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:35:15,896-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 11:35:18,684-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 11:35:25,904-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:35:35,912-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:35:45,920-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 11:35:51,166-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 11:35:55,928-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:35:56,002-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:36:05,936-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:36:15,944-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:36:25,953-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:36:27,639-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:36:33,320-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:36:35,961-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:36:45,969-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:36:55,977-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 11:37:04,113-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 11:37:05,985-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 11:37:10,638-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 11:37:15,993-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:37:26,001-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:37:36,009-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:37:40,587-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:37:46,017-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:37:47,956-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:37:56,025-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:38:06,033-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:38:16,041-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:38:17,061-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 11:38:25,274-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 11:38:26,049-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:38:36,057-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:38:46,065-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:38:53,535-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:38:56,073-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:39:02,592-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:39:06,081-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:39:16,089-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:39:26,097-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 11:39:30,009-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 11:39:36,105-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-07 11:39:39,910-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-07 11:39:46,113-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:39:56,121-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:40:06,129-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:40:06,483-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:40:16,137-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 11:40:17,228-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 11:40:26,145-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:40:36,153-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:40:42,957-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:40:46,161-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:40:54,546-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:40:56,168-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:41:06,176-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:41:16,184-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:41:19,431-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:41:26,192-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 11:41:31,864-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 11:41:36,200-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:41:46,208-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:41:55,905-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:41:56,215-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:42:06,223-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:42:09,182-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:42:16,231-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:42:26,239-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-07 11:42:32,378-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-07 11:42:36,247-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:42:46,255-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 11:42:46,500-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 11:42:56,262-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:43:06,270-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:43:08,853-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:43:16,278-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:43:23,818-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:43:26,286-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:43:36,294-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-07 11:43:45,328-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-07 11:43:46,301-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:43:56,309-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 11:44:01,136-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 11:44:06,317-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:44:16,325-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:44:21,801-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:44:26,333-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:44:36,341-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:44:38,454-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:44:46,348-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:44:56,356-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:44:58,275-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:45:06,364-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 11:45:15,772-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 11:45:16,372-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:45:26,380-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:45:34,749-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:45:36,387-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:45:46,395-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:45:53,090-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:45:56,403-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:46:06,410-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:46:11,223-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:46:16,418-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:46:26,426-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 11:46:30,408-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 11:46:36,433-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:46:46,441-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 11:46:47,697-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 11:46:56,449-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:47:06,456-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:47:07,726-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:47:16,464-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:47:24,172-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:47:26,472-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:47:36,480-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 11:47:45,044-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 11:47:46,488-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:47:56,495-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 11:48:00,647-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 11:48:06,503-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:48:16,510-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:48:22,362-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:48:26,518-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:48:36,526-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:48:37,120-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:48:46,533-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:48:56,541-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-07 11:48:59,680-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-07 11:49:06,549-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:49:13,594-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:49:16,556-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:49:26,564-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:49:36,571-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:49:36,999-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:49:46,579-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:49:50,068-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:49:56,586-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:50:06,594-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:50:14,317-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:50:16,601-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 11:50:26,543-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 11:50:26,608-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:50:36,616-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:50:46,623-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 11:50:51,635-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 11:50:56,631-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-07 11:51:03,015-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-07 11:51:06,638-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:51:16,645-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:51:26,653-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:51:28,953-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:51:36,660-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:51:39,490-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:51:46,668-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:51:56,676-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 11:52:06,271-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 11:52:06,684-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:52:15,965-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:52:16,692-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:52:26,701-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:52:36,709-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:52:43,589-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:52:46,717-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:52:52,439-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:52:56,725-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:53:06,733-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:53:16,741-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 11:53:20,907-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 11:53:26,749-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 11:53:28,913-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 11:53:36,758-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:53:46,765-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:53:56,773-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:53:58,225-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 11:54:05,388-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 11:54:06,781-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:54:16,789-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:54:26,797-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-07 11:54:35,543-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-07 11:54:36,804-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 11:54:41,861-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 11:54:46,812-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:54:56,820-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:55:06,828-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 11:55:12,862-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 11:55:16,836-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:55:18,335-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:55:26,843-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:55:36,851-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:55:46,859-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:55:50,180-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 11:55:54,810-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 11:55:56,867-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:56:06,874-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:56:16,882-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:56:26,890-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 11:56:27,498-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:56:31,283-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:56:36,898-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:56:46,905-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:56:56,913-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:57:04,816-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:57:06,921-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:57:07,757-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:57:16,929-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:57:26,937-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:57:36,944-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-07 11:57:42,134-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:57:44,231-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:57:46,953-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:57:56,961-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:58:06,969-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:58:16,977-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:58:19,452-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 11:58:20,706-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 11:58:26,985-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:58:36,994-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:58:47,002-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:58:56,770-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:58:57,010-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:58:57,179-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 11:59:07,019-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:59:17,027-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:59:27,035-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 11:59:33,653-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 11:59:34,088-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 11:59:37,043-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:59:47,052-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 11:59:57,060-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:00:07,068-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:00:10,127-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:00:11,406-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:00:17,076-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:00:27,085-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:00:37,093-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 3ms
-2013-11-07 12:00:46,603-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 3ms
-No streams
-2013-11-07 12:00:47,101-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:00:48,724-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:00:57,109-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:01:07,118-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:01:17,126-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-07 12:01:23,074-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 12:01:26,042-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 12:01:27,134-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:01:37,142-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:01:47,150-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:01:57,158-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:01:59,549-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:02:03,360-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:02:07,167-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:02:17,175-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:02:27,183-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:02:36,023-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:02:37,191-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 12:02:40,678-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 12:02:47,199-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:02:57,207-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:03:07,215-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 12:03:12,498-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 12:03:17,223-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:03:17,996-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:03:27,231-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:03:37,239-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:03:47,247-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:03:48,971-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-07 12:03:55,313-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-07 12:03:57,255-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:04:07,263-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:04:17,271-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:04:25,445-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:04:27,279-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 12:04:32,632-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 12:04:37,287-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:04:47,295-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:04:57,303-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:05:01,919-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:05:07,311-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:05:09,950-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:05:17,319-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:05:27,327-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:05:37,335-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 12:05:38,394-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:05:47,268-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:05:47,343-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:05:57,350-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:06:07,358-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:06:14,867-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:06:17,367-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:06:24,586-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:06:27,375-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:06:37,383-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:06:47,391-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:06:51,341-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:06:57,399-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:07:01,904-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:07:07,407-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:07:17,415-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:07:27,423-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:07:27,815-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:07:37,431-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 12:07:39,222-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 12:07:47,439-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:07:57,447-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:08:04,289-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:08:07,455-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:08:16,540-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:08:17,463-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:08:27,471-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:08:37,479-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:08:40,763-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:08:47,486-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:08:53,858-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:08:57,494-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:09:07,502-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 12:09:17,238-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 12:09:17,510-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:09:27,519-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 12:09:31,176-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 12:09:37,527-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:09:47,535-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:09:53,711-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:09:57,543-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:10:07,551-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:10:08,494-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:10:17,559-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:10:27,567-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:10:30,185-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:10:37,575-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-07 12:10:45,811-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-07 12:10:47,583-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:10:57,591-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:11:06,659-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:11:07,599-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:11:17,606-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 12:11:23,130-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 12:11:27,614-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:11:37,621-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-07 12:11:43,132-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-07 12:11:47,628-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:11:57,635-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:12:00,448-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:12:07,643-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:12:17,651-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:12:19,607-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:12:27,659-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:12:37,667-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 12:12:37,766-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 12:12:47,675-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:12:56,081-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:12:57,683-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:13:07,691-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:13:15,084-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:13:17,698-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:13:27,706-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:13:32,555-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:13:37,714-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:13:47,721-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:13:52,402-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:13:57,729-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:14:07,737-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-07 12:14:09,028-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-07 12:14:17,745-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:14:27,754-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 12:14:29,720-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 12:14:37,762-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:14:45,503-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:14:47,770-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:14:57,778-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:15:07,038-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:15:07,786-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:15:17,794-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:15:21,977-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:15:27,802-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:15:37,811-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:15:44,357-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:15:47,818-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:15:57,826-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:15:58,451-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:16:07,834-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:16:17,842-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 12:16:21,675-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 12:16:27,850-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 12:16:34,926-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 12:16:37,858-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:16:47,866-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:16:57,873-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:16:58,993-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:17:07,881-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 12:17:11,400-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 12:17:17,889-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:17:27,897-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 12:17:36,311-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 12:17:37,905-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:17:47,873-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:17:47,913-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:17:57,921-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:18:07,928-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:18:13,629-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:18:17,936-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:18:24,347-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:18:27,944-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:18:37,952-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:18:47,960-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:18:50,947-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:18:57,968-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:19:00,821-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:19:07,976-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:19:17,983-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:19:27,991-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 12:19:28,265-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:19:37,295-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:19:37,999-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:19:48,007-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:19:58,015-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:20:05,583-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:20:08,023-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:20:13,769-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:20:18,031-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:20:28,039-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:20:38,047-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 12:20:42,902-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 12:20:48,055-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 12:20:50,244-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 12:20:58,063-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:21:08,071-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:21:18,079-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:21:20,219-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:21:26,717-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:21:28,087-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:21:38,095-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:21:48,103-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:21:57,537-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:21:58,111-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 12:22:03,191-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 12:22:08,119-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:22:18,127-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:22:28,135-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:22:34,855-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:22:38,143-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:22:39,665-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:22:48,151-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:22:58,159-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:23:08,168-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:23:12,173-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:23:16,139-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:23:18,176-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:23:28,184-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:23:38,192-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:23:48,200-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 12:23:49,491-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:23:52,613-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:23:58,208-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:24:08,216-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:24:18,224-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:24:26,809-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:24:28,233-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:24:29,087-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:24:38,241-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:24:48,249-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:24:58,257-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 12:25:04,127-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:25:05,561-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:25:08,265-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:25:18,273-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:25:28,281-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:25:38,289-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:25:41,445-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:25:42,035-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:25:48,298-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:25:58,306-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:26:08,314-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:26:18,322-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:26:18,509-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:26:18,763-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:26:28,330-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:26:38,338-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:26:48,346-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:26:54,983-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 12:26:56,081-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 12:26:58,354-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:27:08,362-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:27:18,370-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:27:28,378-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:27:31,457-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:27:33,399-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:27:38,386-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:27:48,394-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:27:58,402-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:28:07,931-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:28:08,410-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-07 12:28:10,717-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-07 12:28:18,419-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:28:28,427-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:28:38,435-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 12:28:44,405-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:28:48,036-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:28:48,443-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:28:58,451-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:29:08,459-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:29:18,467-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:29:20,880-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:29:25,354-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:29:28,475-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:29:38,483-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:29:48,491-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:29:57,355-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:29:58,500-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:30:02,672-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:30:08,508-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:30:18,516-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:30:28,524-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:30:33,829-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:30:38,532-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:30:39,990-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:30:48,540-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:30:58,549-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:31:08,557-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 12:31:10,303-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-07 12:31:17,307-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-07 12:31:18,565-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:31:28,573-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:31:38,581-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:31:46,777-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:31:48,589-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:31:54,626-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:31:58,597-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:32:08,606-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:32:18,614-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:32:23,251-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:32:28,622-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:32:31,944-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:32:38,630-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:32:48,639-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:32:58,647-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:32:59,725-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:33:08,655-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-07 12:33:09,262-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-07 12:33:18,664-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:33:28,672-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 12:33:36,199-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 12:33:38,680-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:33:46,580-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:33:48,687-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:33:58,695-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:34:08,703-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:34:12,673-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:34:18,711-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-07 12:34:23,898-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-07 12:34:28,719-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:34:38,727-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:34:48,735-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 12:34:49,147-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 12:34:58,743-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:35:01,216-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:35:08,751-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:35:18,759-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:35:25,622-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:35:28,767-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:35:38,534-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:35:38,775-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:35:48,783-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:35:58,791-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:36:02,097-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:36:08,799-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:36:15,852-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:36:18,808-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:36:28,816-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:36:38,571-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:36:38,824-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:36:48,832-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:36:53,170-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:36:58,840-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:37:08,848-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:37:15,045-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:37:18,856-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:37:28,864-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:37:30,488-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:37:38,871-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:37:48,879-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:37:51,519-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:37:58,886-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:38:07,806-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:38:08,894-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:38:18,902-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:38:27,993-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:38:28,910-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:38:38,917-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:38:45,124-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:38:48,925-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:38:58,933-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 12:39:04,467-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 12:39:08,941-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:39:18,949-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:39:22,442-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:39:28,957-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:39:38,965-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:39:40,941-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:39:48,973-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:39:58,981-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:39:59,760-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:40:08,988-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 12:40:17,416-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 12:40:18,996-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:40:29,004-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:40:37,078-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:40:39,012-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:40:49,020-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:40:53,890-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:40:59,028-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:41:09,036-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-07 12:41:14,395-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-07 12:41:19,044-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:41:29,051-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 12:41:30,363-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 12:41:39,059-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:41:49,067-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:41:51,714-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:41:59,075-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:42:06,838-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:42:09,083-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:42:19,091-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:42:29,032-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:42:29,099-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:42:39,107-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 12:42:43,312-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 12:42:49,114-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:42:59,122-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:43:06,350-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:43:09,130-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:43:19,138-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:43:19,786-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:43:29,146-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:43:39,154-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:43:43,668-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:43:49,162-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 12:43:56,260-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 12:43:59,170-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:44:09,178-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:44:19,186-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:44:20,986-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:44:29,194-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:44:32,734-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:44:39,202-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:44:49,210-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:44:58,304-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:44:59,218-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 12:45:09,208-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 12:45:09,226-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:45:19,234-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:45:29,242-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:45:35,622-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:45:39,251-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:45:45,682-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:45:49,259-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:45:59,267-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:46:09,275-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-07 12:46:12,939-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-07 12:46:19,283-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 12:46:22,156-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 12:46:29,291-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:46:39,299-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:46:49,307-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:46:50,258-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:46:58,630-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:46:59,315-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:47:09,323-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:47:19,331-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:47:27,576-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:47:29,339-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 12:47:35,104-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 12:47:39,347-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:47:49,355-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:47:59,363-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:48:04,894-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:48:09,371-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:48:11,578-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:48:19,379-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:48:29,387-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:48:39,395-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:48:42,212-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 12:48:48,052-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 12:48:49,403-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:48:59,411-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:49:09,419-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:49:19,427-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 12:49:19,529-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:49:24,526-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:49:29,435-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:49:39,443-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:49:49,451-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:49:56,848-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:49:59,459-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 12:50:01,000-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 12:50:09,467-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:50:19,475-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:50:29,483-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:50:34,166-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 12:50:37,475-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 12:50:39,491-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:50:49,499-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:50:59,507-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:51:09,515-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 12:51:11,484-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:51:13,949-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:51:19,523-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:51:29,531-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:51:39,538-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:51:48,803-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:51:49,547-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 12:51:50,423-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 12:51:59,555-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:52:09,563-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:52:19,571-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:52:26,121-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:52:26,897-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:52:29,579-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:52:39,587-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:52:49,595-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:52:59,603-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 4ms
-2013-11-07 12:53:03,374-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 4ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 12:53:03,440-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 12:53:09,610-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:53:19,618-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:53:29,626-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:53:39,634-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 12:53:39,845-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:53:40,758-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:53:49,642-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:53:59,649-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:54:09,657-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:54:16,319-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:54:18,076-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:54:19,664-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:54:29,672-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:54:39,680-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:54:49,687-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:54:52,793-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:54:55,394-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:54:59,695-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:55:09,703-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:55:19,710-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 12:55:29,268-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 12:55:29,718-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:55:32,712-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:55:39,726-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:55:49,734-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:55:59,742-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 12:56:05,741-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 12:56:09,749-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:56:10,030-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:56:19,757-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:56:29,765-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:56:39,773-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 12:56:42,215-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:56:47,348-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:56:49,780-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:56:59,788-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:57:09,796-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 12:57:18,691-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 12:57:19,804-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 12:57:24,667-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 12:57:29,814-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:57:39,822-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:57:49,830-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 12:57:55,164-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 12:57:59,838-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 12:58:01,985-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 12:58:09,847-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:58:19,855-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:58:29,863-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 12:58:31,638-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 12:58:39,303-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 12:58:39,872-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:58:49,880-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:58:59,888-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 12:59:08,113-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 12:59:09,896-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 12:59:16,621-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 12:59:19,904-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:59:29,915-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 12:59:39,925-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 12:59:44,587-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 12:59:49,936-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 12:59:53,939-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 12:59:59,946-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:00:09,957-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:00:19,968-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:00:21,060-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:00:29,978-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 13:00:31,257-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 13:00:39,988-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:00:49,999-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 13:00:57,535-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 13:01:00,014-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 13:01:08,575-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 13:01:10,032-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:01:20,051-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:01:30,052-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:01:34,008-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:01:40,054-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 13:01:45,893-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 13:01:50,058-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:02:00,077-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:02:10,080-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 13:02:10,483-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 13:02:20,095-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 13:02:23,211-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 13:02:30,103-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:02:40,111-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:02:46,956-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:02:50,119-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:03:00,127-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 13:03:00,529-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 13:03:10,136-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:03:20,146-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 13:03:23,431-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 13:03:30,158-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 13:03:37,847-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 13:03:40,167-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:03:50,175-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:03:59,904-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:04:00,184-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:04:10,195-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 13:04:15,165-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 13:04:20,205-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:04:30,215-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 13:04:36,378-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 13:04:40,224-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:04:50,233-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 13:04:52,483-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 13:05:00,241-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:05:10,250-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 13:05:12,853-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 13:05:20,258-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 13:05:29,801-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 13:05:30,266-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:05:40,274-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 13:05:49,327-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 13:05:50,282-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:06:00,291-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:06:07,118-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:06:10,302-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:06:20,313-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 13:06:25,801-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 13:06:30,321-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:06:40,329-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:06:44,436-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:06:50,338-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:07:00,346-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:07:02,275-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:07:10,354-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:07:20,363-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:07:21,755-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:07:30,371-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 13:07:38,749-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 13:07:40,379-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:07:50,387-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 13:07:59,073-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 13:08:00,395-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:08:10,403-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-2013-11-07 13:08:15,224-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-No streams
-2013-11-07 13:08:20,411-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:08:30,419-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:08:36,390-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:08:40,427-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:08:50,435-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 13:08:51,697-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 13:09:00,443-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:09:10,451-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:09:13,708-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:09:20,459-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 13:09:28,171-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 13:09:30,467-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:09:40,475-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:09:50,484-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:09:51,026-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:10:00,493-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:10:04,645-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:10:10,502-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:10:20,511-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:10:28,344-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:10:30,523-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:10:40,533-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 13:10:41,119-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 13:10:50,542-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:11:00,551-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 13:11:05,663-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 13:11:10,559-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:11:17,593-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:11:20,568-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:11:30,577-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:11:40,587-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:11:42,980-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:11:50,597-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:11:54,067-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:12:00,608-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:12:10,620-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 13:12:20,299-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 13:12:20,630-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:12:30,541-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:12:30,641-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:12:40,650-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:12:50,658-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 13:12:57,617-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 13:13:00,668-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:13:07,015-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:13:10,677-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:13:20,685-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:13:30,695-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:13:34,934-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:13:40,704-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:13:43,489-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:13:50,713-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:14:00,722-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:14:10,730-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:14:12,252-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 13:14:19,964-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 13:14:20,738-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:14:30,747-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:14:40,755-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:14:49,570-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:14:50,764-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 13:14:56,437-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 13:15:00,773-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:15:10,781-0600 INFO Multiplexer No streams
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 13:15:18,948-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50704] for addition
-2013-11-07 13:15:18,948-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50704] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:50704]
-2013-11-07 13:15:18,948-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:50704]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f4e
-2013-11-07 13:15:18,986-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f4e
-MetaChannel [context: cps-50704, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50704]
-2013-11-07 13:15:18,986-0600 INFO MetaChannel MetaChannel [context: cps-50704, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50704]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-07 13:15:18,986-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-07 13:15:19,047-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-07 13:15:19,047-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 13:15:19,088-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50705] for addition
-2013-11-07 13:15:19,089-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50705] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:50705]
-2013-11-07 13:15:19,089-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:50705]
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f4d
-2013-11-07 13:15:19,107-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f4d
-MetaChannel [context: cps-50705, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50705]
-2013-11-07 13:15:19,107-0600 INFO MetaChannel MetaChannel [context: cps-50705, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50705]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-07 13:15:19,108-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-07 13:15:19,128-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-07 13:15:19,148-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-07 13:15:19,175-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 121.0) submitting urn:job-0-1383759274978-1383759274979: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 13:15:19,175-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 121.0) submitting urn:job-0-1383759274978-1383759274979: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 13:15:19,176-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-07 13:15:19,176-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-07 13:15:19,182-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7574, JOBSTATUS) unregistering (send)
-2013-11-07 13:15:19,183-0600 INFO RequestHandler Handler(tag: 7574, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 13:15:19,188-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50706] for addition
-2013-11-07 13:15:19,189-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50706] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:50706]
-2013-11-07 13:15:19,189-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:50706]
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f4c
-2013-11-07 13:15:19,209-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f4c
-MetaChannel [context: cps-50706, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50706]
-2013-11-07 13:15:19,209-0600 INFO MetaChannel MetaChannel [context: cps-50706, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50706]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-07 13:15:19,209-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-07 13:15:19,229-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-07 13:15:19,270-0600 INFO RequestHandler Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-07 13:15:19,276-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-crush.mcs.anl.gov:0 (quality: 44.0) submitting urn:job-1-1383759274981-1383759274982: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 13:15:19,276-0600 INFO Cpu crush.mcs.anl.gov:0 (quality: 44.0) submitting urn:job-1-1383759274981-1383759274982: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 13:15:19,341-0600 DEBUG Cpu Active workerid=crush.mcs.anl.gov:000000
-Handler(tag: 119, JOBSTATUS) unregistering (send)
-2013-11-07 13:15:19,341-0600 INFO RequestHandler Handler(tag: 119, JOBSTATUS) unregistering (send)
-Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-07 13:15:19,350-0600 INFO RequestHandler Handler(tag: 42566, SUBMITJOB) unregistering (send)
-Avg stream buf: 0
-2013-11-07 13:15:20,795-0600 INFO Multiplexer Avg stream buf: 0
-2013-11-07 13:15:21,209-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 13:15:21,209-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 13:15:21,209-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:15:21,209-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:15:21,209-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-Handler(tag: 7575, JOBSTATUS) unregistering (send)
-2013-11-07 13:15:21,209-0600 INFO RequestHandler Handler(tag: 7575, JOBSTATUS) unregistering (send)
-Channel config: null -> {keepalive=null}
-2013-11-07 13:15:21,209-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50704] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:15:21,209-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50704] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-grind.mcs.anl.gov:0 (quality: 122.0) submitting urn:job-2-1383759274984-1383759274985: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 13:15:21,209-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 122.0) submitting urn:job-2-1383759274984-1383759274985: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-null: Channel shut down
-2013-11-07 13:15:21,210-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-50704, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50704]] binding to NullChannel
-2013-11-07 13:15:21,210-0600 INFO MetaChannel MetaChannel [context: cps-50704, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50704]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50704]
-2013-11-07 13:15:21,210-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50704]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:15:21,210-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 13:15:21,210-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:15:21,210-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 13:15:21,210-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:15:21,210-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 13:15:21,210-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:15:21,210-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 13:15:21,211-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:15:21,211-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 13:15:21,211-0600 INFO ChannelManager Channel exception handled
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:15:21,211-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel config: null -> {keepalive=null}
-2013-11-07 13:15:21,211-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50706] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:15:21,211-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50706] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-07 13:15:21,211-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-50706, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50706]] binding to NullChannel
-2013-11-07 13:15:21,211-0600 INFO MetaChannel MetaChannel [context: cps-50706, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50706]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50706]
-2013-11-07 13:15:21,211-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50706]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:15:21,211-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 13:15:21,211-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:15:21,211-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 13:15:21,211-0600 INFO ChannelManager Channel exception handled
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 13:15:21,216-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 13:15:21,216-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 13:15:21,216-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7576, JOBSTATUS) unregistering (send)
-2013-11-07 13:15:21,216-0600 INFO RequestHandler Handler(tag: 7576, JOBSTATUS) unregistering (send)
-Channel is closed. Removing.
-2013-11-07 13:15:21,217-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 13:15:21,217-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-07 13:15:21,217-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50704] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 13:15:21,217-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50704] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 13:15:21,217-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50704]
-2013-11-07 13:15:21,217-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50704]
-Channel is closed. Removing.
-2013-11-07 13:15:21,217-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 13:15:21,217-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-07 13:15:21,218-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50706] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 13:15:21,218-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50706] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 13:15:21,218-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50706]
-2013-11-07 13:15:21,218-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50706]
-2013-11-07 13:15:21,666-0600 DEBUG Cpu Completed
-crush.mcs.anl.gov:0 jobTerminated
-2013-11-07 13:15:21,666-0600 INFO Cpu crush.mcs.anl.gov:0 jobTerminated
-2013-11-07 13:15:21,666-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:15:21,666-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 120, JOBSTATUS) unregistering (send)
-2013-11-07 13:15:21,666-0600 INFO RequestHandler Handler(tag: 120, JOBSTATUS) unregistering (send)
-2013-11-07 13:15:21,666-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-Channel config: null -> {keepalive=null}
-2013-11-07 13:15:21,667-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-07 13:15:21,667-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-TCPChannel [type: server, contact: 127.0.0.1:50705] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:15:21,667-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50705] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-07 13:15:21,667-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-50705, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50705]] binding to NullChannel
-2013-11-07 13:15:21,667-0600 INFO MetaChannel MetaChannel [context: cps-50705, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50705]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50705]
-2013-11-07 13:15:21,667-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50705]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:15:21,667-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 13:15:21,667-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:15:21,667-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 13:15:21,667-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:15:21,667-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 13:15:21,668-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:15:21,668-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 13:15:21,668-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:15:21,668-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 13:15:21,668-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-07 13:15:21,679-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 13:15:21,679-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-07 13:15:21,680-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50705] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 13:15:21,680-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50705] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 13:15:21,680-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50705]
-2013-11-07 13:15:21,680-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50705]
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 13:15:23,231-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 13:15:23,231-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 13:15:23,231-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 13:15:23,231-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-07 13:15:23,232-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7577, JOBSTATUS) unregistering (send)
-2013-11-07 13:15:23,232-0600 INFO RequestHandler Handler(tag: 7577, JOBSTATUS) unregistering (send)
-2013-11-07 13:15:23,232-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 13:15:26,889-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 13:15:30,809-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 13:15:32,911-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 13:15:40,817-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:15:50,826-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:16:00,835-0600 INFO Multiplexer No streams
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 13:16:00,941-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50709] for addition
-2013-11-07 13:16:00,942-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50709] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:50709]
-2013-11-07 13:16:00,942-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:50709]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f4b
-2013-11-07 13:16:00,976-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f4b
-MetaChannel [context: cps-50709, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50709]
-2013-11-07 13:16:00,976-0600 INFO MetaChannel MetaChannel [context: cps-50709, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50709]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-07 13:16:00,976-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-07 13:16:01,017-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-07 13:16:01,017-0600 INFO Sender Sender TCPChannel queue size: 0
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 13:16:01,056-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50710] for addition
-2013-11-07 13:16:01,056-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50710] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:50710]
-2013-11-07 13:16:01,056-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:50710]
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f4a
-2013-11-07 13:16:01,077-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f4a
-MetaChannel [context: cps-50710, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50710]
-2013-11-07 13:16:01,078-0600 INFO MetaChannel MetaChannel [context: cps-50710, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50710]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-07 13:16:01,078-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-07 13:16:01,098-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-07 13:16:01,118-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-07 13:16:01,128-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 123.0) submitting urn:job-0-1383759274987-1383759274988: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 13:16:01,128-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 123.0) submitting urn:job-0-1383759274987-1383759274988: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 13:16:01,129-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-07 13:16:01,129-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-07 13:16:01,135-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7578, JOBSTATUS) unregistering (send)
-2013-11-07 13:16:01,135-0600 INFO RequestHandler Handler(tag: 7578, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 13:16:01,162-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50711] for addition
-2013-11-07 13:16:01,162-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50711] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:50711]
-2013-11-07 13:16:01,162-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:50711]
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-07 13:16:01,199-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f49
-2013-11-07 13:16:01,199-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f49
-MetaChannel [context: cps-50711, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50711]
-2013-11-07 13:16:01,199-0600 INFO MetaChannel MetaChannel [context: cps-50711, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50711]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-07 13:16:01,199-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-07 13:16:01,229-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-crush.mcs.anl.gov:0 (quality: 45.0) submitting urn:job-1-1383759274990-1383759274991: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 13:16:01,229-0600 INFO Cpu crush.mcs.anl.gov:0 (quality: 45.0) submitting urn:job-1-1383759274990-1383759274991: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 13:16:01,236-0600 DEBUG Cpu Active workerid=crush.mcs.anl.gov:000000
-Handler(tag: 121, JOBSTATUS) unregistering (send)
-2013-11-07 13:16:01,236-0600 INFO RequestHandler Handler(tag: 121, JOBSTATUS) unregistering (send)
-Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-07 13:16:01,240-0600 INFO RequestHandler Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-07 13:16:01,322-0600 INFO RequestHandler Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-07 13:16:03,152-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 13:16:03,152-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:16:03,152-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:16:03,152-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Channel config: null -> {keepalive=null}
-2013-11-07 13:16:03,153-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50709] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:16:03,153-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50709] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 7579, JOBSTATUS) unregistering (send)
-2013-11-07 13:16:03,153-0600 INFO RequestHandler Handler(tag: 7579, JOBSTATUS) unregistering (send)
-2013-11-07 13:16:03,153-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-null: Channel shut down
-2013-11-07 13:16:03,153-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-50709, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50709]] binding to NullChannel
-2013-11-07 13:16:03,153-0600 INFO MetaChannel MetaChannel [context: cps-50709, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50709]] binding to NullChannel
-grind.mcs.anl.gov:0 (quality: 124.0) submitting urn:job-2-1383759274993-1383759274994: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 13:16:03,153-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 124.0) submitting urn:job-2-1383759274993-1383759274994: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50709]
-2013-11-07 13:16:03,153-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50709]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:16:03,153-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 13:16:03,154-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:16:03,154-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 13:16:03,154-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:16:03,154-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 13:16:03,154-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:16:03,154-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 13:16:03,154-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:16:03,154-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 13:16:03,154-0600 INFO ChannelManager Channel exception handled
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:16:03,154-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel config: null -> {keepalive=null}
-2013-11-07 13:16:03,154-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50711] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:16:03,154-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50711] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-07 13:16:03,155-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-50711, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50711]] binding to NullChannel
-2013-11-07 13:16:03,155-0600 INFO MetaChannel MetaChannel [context: cps-50711, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50711]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50711]
-2013-11-07 13:16:03,155-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50711]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:16:03,155-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 13:16:03,155-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:16:03,155-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 13:16:03,155-0600 INFO ChannelManager Channel exception handled
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 13:16:03,159-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 13:16:03,159-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 13:16:03,160-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7580, JOBSTATUS) unregistering (send)
-2013-11-07 13:16:03,160-0600 INFO RequestHandler Handler(tag: 7580, JOBSTATUS) unregistering (send)
-Channel is closed. Removing.
-2013-11-07 13:16:03,168-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 13:16:03,168-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-07 13:16:03,168-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50711] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 13:16:03,168-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50711] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 13:16:03,168-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50711]
-2013-11-07 13:16:03,168-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50711]
-Channel is closed. Removing.
-2013-11-07 13:16:03,168-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 13:16:03,168-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-07 13:16:03,168-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50709] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 13:16:03,168-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50709] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 13:16:03,168-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50709]
-2013-11-07 13:16:03,168-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50709]
-2013-11-07 13:16:03,253-0600 DEBUG Cpu Completed
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:16:03,253-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-crush.mcs.anl.gov:0 jobTerminated
-2013-11-07 13:16:03,253-0600 INFO Cpu crush.mcs.anl.gov:0 jobTerminated
-Channel config: null -> {}
-2013-11-07 13:16:03,253-0600 INFO AbstractCoasterChannel Channel config: null -> {}
-2013-11-07 13:16:03,253-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-TCPChannel [type: server, contact: 127.0.0.1:50710] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:16:03,253-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50710] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 122, JOBSTATUS) unregistering (send)
-2013-11-07 13:16:03,253-0600 INFO RequestHandler Handler(tag: 122, JOBSTATUS) unregistering (send)
-null: Channel shut down
-2013-11-07 13:16:03,253-0600 INFO AbstractTCPChannel null: Channel shut down
-2013-11-07 13:16:03,253-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-MetaChannel [context: cps-50710, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50710]] binding to NullChannel
-2013-11-07 13:16:03,253-0600 INFO MetaChannel MetaChannel [context: cps-50710, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50710]] binding to NullChannel
-2013-11-07 13:16:03,253-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50710]
-2013-11-07 13:16:03,253-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50710]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:16:03,253-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 13:16:03,253-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:16:03,253-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 13:16:03,254-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:16:03,254-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 13:16:03,254-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:16:03,254-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 13:16:03,254-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:16:03,254-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 13:16:03,254-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-07 13:16:03,269-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 13:16:03,269-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {}
-2013-11-07 13:16:03,269-0600 INFO AbstractCoasterChannel Channel config: null -> {}
-TCPChannel [type: server, contact: 127.0.0.1:50710] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 13:16:03,269-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50710] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 13:16:03,269-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50710]
-2013-11-07 13:16:03,269-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50710]
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:16:04,206-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 13:16:05,175-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 13:16:05,175-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 13:16:05,175-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 13:16:05,175-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-07 13:16:05,175-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7581, JOBSTATUS) unregistering (send)
-2013-11-07 13:16:05,175-0600 INFO RequestHandler Handler(tag: 7581, JOBSTATUS) unregistering (send)
-2013-11-07 13:16:05,175-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 13:16:09,385-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 13:16:10,850-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:16:20,858-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:16:30,870-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:16:40,880-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:16:41,525-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:16:45,859-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:16:50,888-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:17:00,897-0600 INFO Multiplexer No streams
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 13:17:09,461-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50714] for addition
-2013-11-07 13:17:09,462-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50714] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:50714]
-2013-11-07 13:17:09,462-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:50714]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f48
-2013-11-07 13:17:09,484-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f48
-MetaChannel [context: cps-50714, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50714]
-2013-11-07 13:17:09,484-0600 INFO MetaChannel MetaChannel [context: cps-50714, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50714]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-07 13:17:09,484-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-07 13:17:09,525-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-07 13:17:09,525-0600 INFO Sender Sender TCPChannel queue size: 0
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 13:17:09,564-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50715] for addition
-2013-11-07 13:17:09,564-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50715] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:50715]
-2013-11-07 13:17:09,564-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:50715]
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f47
-2013-11-07 13:17:09,566-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f47
-MetaChannel [context: cps-50715, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50715]
-2013-11-07 13:17:09,566-0600 INFO MetaChannel MetaChannel [context: cps-50715, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50715]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-07 13:17:09,566-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-07 13:17:09,607-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-07 13:17:09,607-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 13:17:09,648-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50716] for addition
-2013-11-07 13:17:09,648-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50716] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:50716]
-2013-11-07 13:17:09,648-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:50716]
-2013-11-07 13:17:09,654-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 125.0) submitting urn:job-0-1383759274996-1383759274997: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 13:17:09,654-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 125.0) submitting urn:job-0-1383759274996-1383759274997: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 13:17:09,655-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-07 13:17:09,655-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-07 13:17:09,661-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7582, JOBSTATUS) unregistering (send)
-2013-11-07 13:17:09,661-0600 INFO RequestHandler Handler(tag: 7582, JOBSTATUS) unregistering (send)
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-07 13:17:09,688-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f46
-2013-11-07 13:17:09,688-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f46
-MetaChannel [context: cps-50716, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50716]
-2013-11-07 13:17:09,688-0600 INFO MetaChannel MetaChannel [context: cps-50716, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50716]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-07 13:17:09,688-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-07 13:17:09,705-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-crush.mcs.anl.gov:0 (quality: 46.0) submitting urn:job-1-1383759274999-1383759275000: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 13:17:09,705-0600 INFO Cpu crush.mcs.anl.gov:0 (quality: 46.0) submitting urn:job-1-1383759274999-1383759275000: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-07 13:17:09,709-0600 INFO RequestHandler Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-07 13:17:09,713-0600 DEBUG Cpu Active workerid=crush.mcs.anl.gov:000000
-Handler(tag: 123, JOBSTATUS) unregistering (send)
-2013-11-07 13:17:09,713-0600 INFO RequestHandler Handler(tag: 123, JOBSTATUS) unregistering (send)
-Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-07 13:17:09,791-0600 INFO RequestHandler Handler(tag: 42566, SUBMITJOB) unregistering (send)
-Avg stream buf: 0
-2013-11-07 13:17:10,914-0600 INFO Multiplexer Avg stream buf: 0
-2013-11-07 13:17:11,686-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 13:17:11,686-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 13:17:11,687-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:17:11,687-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:17:11,687-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-Handler(tag: 7583, JOBSTATUS) unregistering (send)
-2013-11-07 13:17:11,687-0600 INFO RequestHandler Handler(tag: 7583, JOBSTATUS) unregistering (send)
-Channel config: null -> {keepalive=null}
-2013-11-07 13:17:11,687-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-grind.mcs.anl.gov:0 (quality: 126.0) submitting urn:job-2-1383759275002-1383759275003: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 13:17:11,687-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 126.0) submitting urn:job-2-1383759275002-1383759275003: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-TCPChannel [type: server, contact: 127.0.0.1:50714] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:17:11,687-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50714] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-07 13:17:11,688-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-50714, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50714]] binding to NullChannel
-2013-11-07 13:17:11,688-0600 INFO MetaChannel MetaChannel [context: cps-50714, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50714]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50714]
-2013-11-07 13:17:11,688-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50714]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:17:11,688-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 13:17:11,688-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:17:11,688-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 13:17:11,688-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:17:11,688-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 13:17:11,688-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:17:11,688-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 13:17:11,688-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:17:11,688-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 13:17:11,688-0600 INFO ChannelManager Channel exception handled
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:17:11,689-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel config: null -> {keepalive=null}
-2013-11-07 13:17:11,689-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50716] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:17:11,689-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50716] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-07 13:17:11,689-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-50716, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50716]] binding to NullChannel
-2013-11-07 13:17:11,689-0600 INFO MetaChannel MetaChannel [context: cps-50716, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50716]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50716]
-2013-11-07 13:17:11,689-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50716]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:17:11,689-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 13:17:11,689-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:17:11,689-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 13:17:11,689-0600 INFO ChannelManager Channel exception handled
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 13:17:11,693-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 13:17:11,694-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 13:17:11,694-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7584, JOBSTATUS) unregistering (send)
-2013-11-07 13:17:11,694-0600 INFO RequestHandler Handler(tag: 7584, JOBSTATUS) unregistering (send)
-Channel is closed. Removing.
-2013-11-07 13:17:11,697-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 13:17:11,697-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-07 13:17:11,697-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50716] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 13:17:11,697-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50716] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 13:17:11,697-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50716]
-2013-11-07 13:17:11,697-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50716]
-Channel is closed. Removing.
-2013-11-07 13:17:11,697-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 13:17:11,697-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-07 13:17:11,697-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50714] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 13:17:11,697-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50714] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 13:17:11,697-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50714]
-2013-11-07 13:17:11,697-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50714]
-2013-11-07 13:17:11,738-0600 DEBUG Cpu Completed
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:17:11,738-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-crush.mcs.anl.gov:0 jobTerminated
-2013-11-07 13:17:11,738-0600 INFO Cpu crush.mcs.anl.gov:0 jobTerminated
-Channel config: null -> {}
-2013-11-07 13:17:11,738-0600 INFO AbstractCoasterChannel Channel config: null -> {}
-2013-11-07 13:17:11,738-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-TCPChannel [type: server, contact: 127.0.0.1:50715] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:17:11,738-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50715] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 124, JOBSTATUS) unregistering (send)
-2013-11-07 13:17:11,738-0600 INFO RequestHandler Handler(tag: 124, JOBSTATUS) unregistering (send)
-2013-11-07 13:17:11,738-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-null: Channel shut down
-2013-11-07 13:17:11,739-0600 INFO AbstractTCPChannel null: Channel shut down
-2013-11-07 13:17:11,739-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-MetaChannel [context: cps-50715, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50715]] binding to NullChannel
-2013-11-07 13:17:11,739-0600 INFO MetaChannel MetaChannel [context: cps-50715, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50715]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50715]
-2013-11-07 13:17:11,739-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50715]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:17:11,739-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 13:17:11,739-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:17:11,739-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 13:17:11,739-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:17:11,739-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 13:17:11,739-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:17:11,739-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 13:17:11,739-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:17:11,739-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 13:17:11,740-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-07 13:17:11,758-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 13:17:11,758-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {}
-2013-11-07 13:17:11,758-0600 INFO AbstractCoasterChannel Channel config: null -> {}
-TCPChannel [type: server, contact: 127.0.0.1:50715] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 13:17:11,758-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50715] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 13:17:11,758-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50715]
-2013-11-07 13:17:11,758-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50715]
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 13:17:13,710-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 13:17:13,710-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 13:17:13,710-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 13:17:13,710-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-07 13:17:13,710-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7585, JOBSTATUS) unregistering (send)
-2013-11-07 13:17:13,710-0600 INFO RequestHandler Handler(tag: 7585, JOBSTATUS) unregistering (send)
-2013-11-07 13:17:13,711-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:17:18,842-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131519.128, now=131107-131719.948, channel=TCPChannel [type: server, contact: 127.0.0.1:50704]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:17:19,948-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131519.128, now=131107-131719.948, channel=TCPChannel [type: server, contact: 127.0.0.1:50704]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 13:17:19,948-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131519.128, now=131107-131719.948, channel=TCPChannel [type: server, contact: 127.0.0.1:50704]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:17:19,948-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131519.128, now=131107-131719.948, channel=TCPChannel [type: server, contact: 127.0.0.1:50704]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 13:17:19,948-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131519.128, now=131107-131719.948, channel=TCPChannel [type: server, contact: 127.0.0.1:50704]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:17:19,948-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131519.128, now=131107-131719.948, channel=TCPChannel [type: server, contact: 127.0.0.1:50704]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 13:17:19,948-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131519.128, now=131107-131719.948, channel=TCPChannel [type: server, contact: 127.0.0.1:50704]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:17:19,949-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131519.128, now=131107-131719.948, channel=TCPChannel [type: server, contact: 127.0.0.1:50704]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 13:17:19,949-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131519.128, now=131107-131719.948, channel=TCPChannel [type: server, contact: 127.0.0.1:50704]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:17:19,949-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131519.128, now=131107-131719.948, channel=TCPChannel [type: server, contact: 127.0.0.1:50704]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-07 13:17:19,949-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50704] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131519.128, now=131107-131719.948, channel=TCPChannel [type: server, contact: 127.0.0.1:50704]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:17:19,949-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50704] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131519.128, now=131107-131719.948, channel=TCPChannel [type: server, contact: 127.0.0.1:50704]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 13:17:19,949-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131519.229, now=131107-131720.088, channel=TCPChannel [type: server, contact: 127.0.0.1:50705]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:17:20,088-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131519.229, now=131107-131720.088, channel=TCPChannel [type: server, contact: 127.0.0.1:50705]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 13:17:20,088-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131519.229, now=131107-131720.088, channel=TCPChannel [type: server, contact: 127.0.0.1:50705]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:17:20,088-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131519.229, now=131107-131720.088, channel=TCPChannel [type: server, contact: 127.0.0.1:50705]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 13:17:20,088-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131519.229, now=131107-131720.088, channel=TCPChannel [type: server, contact: 127.0.0.1:50705]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:17:20,088-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131519.229, now=131107-131720.088, channel=TCPChannel [type: server, contact: 127.0.0.1:50705]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 13:17:20,089-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131519.229, now=131107-131720.088, channel=TCPChannel [type: server, contact: 127.0.0.1:50705]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:17:20,089-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131519.229, now=131107-131720.088, channel=TCPChannel [type: server, contact: 127.0.0.1:50705]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 13:17:20,089-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131519.229, now=131107-131720.088, channel=TCPChannel [type: server, contact: 127.0.0.1:50705]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:17:20,089-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131519.229, now=131107-131720.088, channel=TCPChannel [type: server, contact: 127.0.0.1:50705]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-07 13:17:20,089-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50705] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131519.229, now=131107-131720.088, channel=TCPChannel [type: server, contact: 127.0.0.1:50705]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:17:20,089-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50705] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131519.229, now=131107-131720.088, channel=TCPChannel [type: server, contact: 127.0.0.1:50705]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 13:17:20,089-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131519.350, now=131107-131720.188, channel=TCPChannel [type: server, contact: 127.0.0.1:50706]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:17:20,188-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131519.350, now=131107-131720.188, channel=TCPChannel [type: server, contact: 127.0.0.1:50706]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 13:17:20,188-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131519.350, now=131107-131720.188, channel=TCPChannel [type: server, contact: 127.0.0.1:50706]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:17:20,188-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131519.350, now=131107-131720.188, channel=TCPChannel [type: server, contact: 127.0.0.1:50706]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-07 13:17:20,189-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50706] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131519.350, now=131107-131720.188, channel=TCPChannel [type: server, contact: 127.0.0.1:50706]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:17:20,189-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50706] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131519.350, now=131107-131720.188, channel=TCPChannel [type: server, contact: 127.0.0.1:50706]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 13:17:20,189-0600 INFO ChannelManager Channel already shut down
-No streams
-2013-11-07 13:17:20,925-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 13:17:22,333-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 13:17:30,933-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:17:40,942-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:17:50,952-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 13:17:56,161-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:17:58,807-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:18:00,962-0600 INFO Multiplexer No streams
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131601.098, now=131107-131801.942, channel=TCPChannel [type: server, contact: 127.0.0.1:50709]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:18:01,942-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131601.098, now=131107-131801.942, channel=TCPChannel [type: server, contact: 127.0.0.1:50709]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 13:18:01,942-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131601.098, now=131107-131801.942, channel=TCPChannel [type: server, contact: 127.0.0.1:50709]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:18:01,942-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131601.098, now=131107-131801.942, channel=TCPChannel [type: server, contact: 127.0.0.1:50709]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 13:18:01,942-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131601.098, now=131107-131801.942, channel=TCPChannel [type: server, contact: 127.0.0.1:50709]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:18:01,942-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131601.098, now=131107-131801.942, channel=TCPChannel [type: server, contact: 127.0.0.1:50709]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 13:18:01,942-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131601.098, now=131107-131801.942, channel=TCPChannel [type: server, contact: 127.0.0.1:50709]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:18:01,942-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131601.098, now=131107-131801.942, channel=TCPChannel [type: server, contact: 127.0.0.1:50709]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 13:18:01,942-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131601.098, now=131107-131801.942, channel=TCPChannel [type: server, contact: 127.0.0.1:50709]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:18:01,942-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131601.098, now=131107-131801.942, channel=TCPChannel [type: server, contact: 127.0.0.1:50709]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-07 13:18:01,943-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50709] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131601.098, now=131107-131801.942, channel=TCPChannel [type: server, contact: 127.0.0.1:50709]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:18:01,943-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50709] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131601.098, now=131107-131801.942, channel=TCPChannel [type: server, contact: 127.0.0.1:50709]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 13:18:01,943-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131601.199, now=131107-131802.056, channel=TCPChannel [type: server, contact: 127.0.0.1:50710]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:18:02,056-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131601.199, now=131107-131802.056, channel=TCPChannel [type: server, contact: 127.0.0.1:50710]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 13:18:02,056-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131601.199, now=131107-131802.056, channel=TCPChannel [type: server, contact: 127.0.0.1:50710]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:18:02,056-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131601.199, now=131107-131802.056, channel=TCPChannel [type: server, contact: 127.0.0.1:50710]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 13:18:02,056-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131601.199, now=131107-131802.056, channel=TCPChannel [type: server, contact: 127.0.0.1:50710]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:18:02,056-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131601.199, now=131107-131802.056, channel=TCPChannel [type: server, contact: 127.0.0.1:50710]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 13:18:02,056-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131601.199, now=131107-131802.056, channel=TCPChannel [type: server, contact: 127.0.0.1:50710]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:18:02,056-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131601.199, now=131107-131802.056, channel=TCPChannel [type: server, contact: 127.0.0.1:50710]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 13:18:02,056-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131601.199, now=131107-131802.056, channel=TCPChannel [type: server, contact: 127.0.0.1:50710]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:18:02,056-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131601.199, now=131107-131802.056, channel=TCPChannel [type: server, contact: 127.0.0.1:50710]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {}
-2013-11-07 13:18:02,056-0600 INFO AbstractCoasterChannel Channel config: null -> {}
-TCPChannel [type: server, contact: 127.0.0.1:50710] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131601.199, now=131107-131802.056, channel=TCPChannel [type: server, contact: 127.0.0.1:50710]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:18:02,056-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50710] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131601.199, now=131107-131802.056, channel=TCPChannel [type: server, contact: 127.0.0.1:50710]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 13:18:02,056-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131601.322, now=131107-131802.162, channel=TCPChannel [type: server, contact: 127.0.0.1:50711]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:18:02,162-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131601.322, now=131107-131802.162, channel=TCPChannel [type: server, contact: 127.0.0.1:50711]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 13:18:02,163-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131601.322, now=131107-131802.162, channel=TCPChannel [type: server, contact: 127.0.0.1:50711]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:18:02,163-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131601.322, now=131107-131802.162, channel=TCPChannel [type: server, contact: 127.0.0.1:50711]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-07 13:18:02,163-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50711] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131601.322, now=131107-131802.162, channel=TCPChannel [type: server, contact: 127.0.0.1:50711]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:18:02,163-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50711] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131601.322, now=131107-131802.162, channel=TCPChannel [type: server, contact: 127.0.0.1:50711]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 13:18:02,163-0600 INFO ChannelManager Channel already shut down
-No streams
-2013-11-07 13:18:10,973-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:18:20,982-0600 INFO Multiplexer No streams
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 13:18:29,168-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50717] for addition
-2013-11-07 13:18:29,168-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50717] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:50717]
-2013-11-07 13:18:29,168-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:50717]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f45
-2013-11-07 13:18:29,209-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f45
-MetaChannel [context: cps-50717, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50717]
-2013-11-07 13:18:29,209-0600 INFO MetaChannel MetaChannel [context: cps-50717, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50717]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-07 13:18:29,209-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-07 13:18:29,249-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-07 13:18:29,249-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 13:18:29,288-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50718] for addition
-2013-11-07 13:18:29,288-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50718] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:50718]
-2013-11-07 13:18:29,288-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:50718]
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f44
-2013-11-07 13:18:29,290-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f44
-MetaChannel [context: cps-50718, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50718]
-2013-11-07 13:18:29,290-0600 INFO MetaChannel MetaChannel [context: cps-50718, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50718]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-07 13:18:29,290-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-07 13:18:29,331-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-07 13:18:29,331-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-07 13:18:29,354-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 127.0) submitting urn:job-0-1383759275005-1383759275006: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 13:18:29,355-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 127.0) submitting urn:job-0-1383759275005-1383759275006: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 13:18:29,356-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-07 13:18:29,356-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-07 13:18:29,360-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7586, JOBSTATUS) unregistering (send)
-2013-11-07 13:18:29,360-0600 INFO RequestHandler Handler(tag: 7586, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 13:18:29,376-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50719] for addition
-2013-11-07 13:18:29,376-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:50719] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:50719]
-2013-11-07 13:18:29,376-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:50719]
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f43
-2013-11-07 13:18:29,391-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f43
-MetaChannel [context: cps-50719, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50719]
-2013-11-07 13:18:29,391-0600 INFO MetaChannel MetaChannel [context: cps-50719, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:50719]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-07 13:18:29,391-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-07 13:18:29,432-0600 INFO RequestHandler Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-07 13:18:29,432-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-07 13:18:29,456-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-crush.mcs.anl.gov:0 (quality: 47.0) submitting urn:job-1-1383759275008-1383759275009: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 13:18:29,456-0600 INFO Cpu crush.mcs.anl.gov:0 (quality: 47.0) submitting urn:job-1-1383759275008-1383759275009: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 13:18:29,463-0600 DEBUG Cpu Active workerid=crush.mcs.anl.gov:000000
-Handler(tag: 125, JOBSTATUS) unregistering (send)
-2013-11-07 13:18:29,463-0600 INFO RequestHandler Handler(tag: 125, JOBSTATUS) unregistering (send)
-Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-07 13:18:29,513-0600 INFO RequestHandler Handler(tag: 42566, SUBMITJOB) unregistering (send)
-Avg stream buf: 0
-2013-11-07 13:18:30,997-0600 INFO Multiplexer Avg stream buf: 0
-2013-11-07 13:18:31,382-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 13:18:31,383-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:18:31,383-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:18:31,383-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Channel config: null -> {keepalive=null}
-2013-11-07 13:18:31,383-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50717] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:18:31,383-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50717] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:18:31,383-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-Handler(tag: 7587, JOBSTATUS) unregistering (send)
-2013-11-07 13:18:31,383-0600 INFO RequestHandler Handler(tag: 7587, JOBSTATUS) unregistering (send)
-grind.mcs.anl.gov:0 (quality: 128.0) submitting urn:job-2-1383759275011-1383759275012: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 13:18:31,383-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 128.0) submitting urn:job-2-1383759275011-1383759275012: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-null: Channel shut down
-2013-11-07 13:18:31,383-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-50717, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50717]] binding to NullChannel
-2013-11-07 13:18:31,384-0600 INFO MetaChannel MetaChannel [context: cps-50717, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50717]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50717]
-2013-11-07 13:18:31,384-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50717]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:18:31,384-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 13:18:31,384-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:18:31,384-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 13:18:31,384-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:18:31,385-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 13:18:31,385-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:18:31,385-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 13:18:31,385-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:18:31,385-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 13:18:31,385-0600 INFO ChannelManager Channel exception handled
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:18:31,385-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel config: null -> {keepalive=null}
-2013-11-07 13:18:31,385-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50719] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:18:31,385-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50719] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-07 13:18:31,385-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-50719, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50719]] binding to NullChannel
-2013-11-07 13:18:31,385-0600 INFO MetaChannel MetaChannel [context: cps-50719, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50719]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50719]
-2013-11-07 13:18:31,385-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50719]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:18:31,385-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 13:18:31,386-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:18:31,386-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 13:18:31,386-0600 INFO ChannelManager Channel exception handled
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 13:18:31,390-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 13:18:31,391-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 13:18:31,391-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7588, JOBSTATUS) unregistering (send)
-2013-11-07 13:18:31,391-0600 INFO RequestHandler Handler(tag: 7588, JOBSTATUS) unregistering (send)
-Channel is closed. Removing.
-2013-11-07 13:18:31,398-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 13:18:31,399-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-07 13:18:31,399-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50719] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 13:18:31,399-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50719] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 13:18:31,399-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50719]
-2013-11-07 13:18:31,399-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50719]
-Channel is closed. Removing.
-2013-11-07 13:18:31,399-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 13:18:31,399-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-07 13:18:31,399-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50717] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 13:18:31,399-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50717] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 13:18:31,399-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50717]
-2013-11-07 13:18:31,399-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50717]
-2013-11-07 13:18:31,486-0600 DEBUG Cpu Completed
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:18:31,486-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-crush.mcs.anl.gov:0 jobTerminated
-2013-11-07 13:18:31,486-0600 INFO Cpu crush.mcs.anl.gov:0 jobTerminated
-Channel config: null -> {}
-2013-11-07 13:18:31,486-0600 INFO AbstractCoasterChannel Channel config: null -> {}
-2013-11-07 13:18:31,486-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-TCPChannel [type: server, contact: 127.0.0.1:50718] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:18:31,486-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50718] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:18:31,486-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-Handler(tag: 126, JOBSTATUS) unregistering (send)
-2013-11-07 13:18:31,486-0600 INFO RequestHandler Handler(tag: 126, JOBSTATUS) unregistering (send)
-2013-11-07 13:18:31,486-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-null: Channel shut down
-2013-11-07 13:18:31,486-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-50718, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50718]] binding to NullChannel
-2013-11-07 13:18:31,487-0600 INFO MetaChannel MetaChannel [context: cps-50718, boundTo: TCPChannel [type: server, contact: 127.0.0.1:50718]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50718]
-2013-11-07 13:18:31,487-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:50718]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:18:31,487-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 13:18:31,487-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:18:31,487-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 13:18:31,487-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:18:31,487-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 13:18:31,487-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:18:31,487-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 13:18:31,487-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:18:31,487-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 13:18:31,487-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-07 13:18:31,499-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 13:18:31,499-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {}
-2013-11-07 13:18:31,499-0600 INFO AbstractCoasterChannel Channel config: null -> {}
-TCPChannel [type: server, contact: 127.0.0.1:50718] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 13:18:31,500-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50718] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 13:18:31,500-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50718]
-2013-11-07 13:18:31,500-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:50718]
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 13:18:33,409-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 13:18:33,409-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 13:18:33,409-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 13:18:33,409-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Handler(tag: 7589, JOBSTATUS) unregistering (send)
-2013-11-07 13:18:33,409-0600 INFO RequestHandler Handler(tag: 7589, JOBSTATUS) unregistering (send)
-2013-11-07 13:18:33,409-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-2013-11-07 13:18:33,410-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:18:33,478-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 13:18:35,281-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 13:18:41,006-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:18:51,017-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:19:01,026-0600 INFO Multiplexer No streams
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131709.606, now=131107-131910.461, channel=TCPChannel [type: server, contact: 127.0.0.1:50714]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:19:10,461-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131709.606, now=131107-131910.461, channel=TCPChannel [type: server, contact: 127.0.0.1:50714]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 13:19:10,461-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131709.606, now=131107-131910.461, channel=TCPChannel [type: server, contact: 127.0.0.1:50714]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:19:10,461-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131709.606, now=131107-131910.461, channel=TCPChannel [type: server, contact: 127.0.0.1:50714]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 13:19:10,461-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131709.606, now=131107-131910.461, channel=TCPChannel [type: server, contact: 127.0.0.1:50714]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:19:10,461-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131709.606, now=131107-131910.461, channel=TCPChannel [type: server, contact: 127.0.0.1:50714]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 13:19:10,461-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131709.606, now=131107-131910.461, channel=TCPChannel [type: server, contact: 127.0.0.1:50714]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:19:10,461-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131709.606, now=131107-131910.461, channel=TCPChannel [type: server, contact: 127.0.0.1:50714]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 13:19:10,461-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131709.606, now=131107-131910.461, channel=TCPChannel [type: server, contact: 127.0.0.1:50714]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:19:10,461-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131709.606, now=131107-131910.461, channel=TCPChannel [type: server, contact: 127.0.0.1:50714]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-07 13:19:10,461-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50714] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131709.606, now=131107-131910.461, channel=TCPChannel [type: server, contact: 127.0.0.1:50714]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:19:10,461-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50714] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131709.606, now=131107-131910.461, channel=TCPChannel [type: server, contact: 127.0.0.1:50714]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 13:19:10,462-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131709.688, now=131107-131910.564, channel=TCPChannel [type: server, contact: 127.0.0.1:50715]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:19:10,564-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131709.688, now=131107-131910.564, channel=TCPChannel [type: server, contact: 127.0.0.1:50715]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 13:19:10,564-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131709.688, now=131107-131910.564, channel=TCPChannel [type: server, contact: 127.0.0.1:50715]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:19:10,564-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131709.688, now=131107-131910.564, channel=TCPChannel [type: server, contact: 127.0.0.1:50715]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 13:19:10,564-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131709.688, now=131107-131910.564, channel=TCPChannel [type: server, contact: 127.0.0.1:50715]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:19:10,564-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131709.688, now=131107-131910.564, channel=TCPChannel [type: server, contact: 127.0.0.1:50715]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 13:19:10,564-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131709.688, now=131107-131910.564, channel=TCPChannel [type: server, contact: 127.0.0.1:50715]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:19:10,564-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131709.688, now=131107-131910.564, channel=TCPChannel [type: server, contact: 127.0.0.1:50715]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 13:19:10,564-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131709.688, now=131107-131910.564, channel=TCPChannel [type: server, contact: 127.0.0.1:50715]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:19:10,564-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131709.688, now=131107-131910.564, channel=TCPChannel [type: server, contact: 127.0.0.1:50715]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {}
-2013-11-07 13:19:10,564-0600 INFO AbstractCoasterChannel Channel config: null -> {}
-TCPChannel [type: server, contact: 127.0.0.1:50715] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131709.688, now=131107-131910.564, channel=TCPChannel [type: server, contact: 127.0.0.1:50715]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:19:10,564-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50715] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131709.688, now=131107-131910.564, channel=TCPChannel [type: server, contact: 127.0.0.1:50715]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 13:19:10,564-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131709.791, now=131107-131910.648, channel=TCPChannel [type: server, contact: 127.0.0.1:50716]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:19:10,648-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131709.791, now=131107-131910.648, channel=TCPChannel [type: server, contact: 127.0.0.1:50716]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 13:19:10,648-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131709.791, now=131107-131910.648, channel=TCPChannel [type: server, contact: 127.0.0.1:50716]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:19:10,648-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131709.791, now=131107-131910.648, channel=TCPChannel [type: server, contact: 127.0.0.1:50716]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-07 13:19:10,648-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50716] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131709.791, now=131107-131910.648, channel=TCPChannel [type: server, contact: 127.0.0.1:50716]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:19:10,649-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50716] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131709.791, now=131107-131910.648, channel=TCPChannel [type: server, contact: 127.0.0.1:50716]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 13:19:10,649-0600 INFO ChannelManager Channel already shut down
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:19:10,796-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:19:11,034-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:19:11,755-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:19:21,041-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:19:31,049-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:19:41,057-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:19:48,114-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:19:48,229-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:19:51,064-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:20:01,072-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:20:11,080-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:20:21,089-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 13:20:24,703-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:20:25,432-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131829.330, now=131107-132030.168, channel=TCPChannel [type: server, contact: 127.0.0.1:50717]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:20:30,168-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131829.330, now=131107-132030.168, channel=TCPChannel [type: server, contact: 127.0.0.1:50717]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 13:20:30,168-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131829.330, now=131107-132030.168, channel=TCPChannel [type: server, contact: 127.0.0.1:50717]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:20:30,168-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131829.330, now=131107-132030.168, channel=TCPChannel [type: server, contact: 127.0.0.1:50717]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 13:20:30,168-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131829.330, now=131107-132030.168, channel=TCPChannel [type: server, contact: 127.0.0.1:50717]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:20:30,168-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131829.330, now=131107-132030.168, channel=TCPChannel [type: server, contact: 127.0.0.1:50717]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 13:20:30,168-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131829.330, now=131107-132030.168, channel=TCPChannel [type: server, contact: 127.0.0.1:50717]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:20:30,168-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131829.330, now=131107-132030.168, channel=TCPChannel [type: server, contact: 127.0.0.1:50717]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 13:20:30,168-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131829.330, now=131107-132030.168, channel=TCPChannel [type: server, contact: 127.0.0.1:50717]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:20:30,168-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131829.330, now=131107-132030.168, channel=TCPChannel [type: server, contact: 127.0.0.1:50717]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-07 13:20:30,168-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50717] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131829.330, now=131107-132030.168, channel=TCPChannel [type: server, contact: 127.0.0.1:50717]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:20:30,168-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50717] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131829.330, now=131107-132030.168, channel=TCPChannel [type: server, contact: 127.0.0.1:50717]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 13:20:30,168-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131829.432, now=131107-132030.288, channel=TCPChannel [type: server, contact: 127.0.0.1:50718]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:20:30,288-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131829.432, now=131107-132030.288, channel=TCPChannel [type: server, contact: 127.0.0.1:50718]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 13:20:30,289-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131829.432, now=131107-132030.288, channel=TCPChannel [type: server, contact: 127.0.0.1:50718]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:20:30,289-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131829.432, now=131107-132030.288, channel=TCPChannel [type: server, contact: 127.0.0.1:50718]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 13:20:30,289-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131829.432, now=131107-132030.288, channel=TCPChannel [type: server, contact: 127.0.0.1:50718]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:20:30,289-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131829.432, now=131107-132030.288, channel=TCPChannel [type: server, contact: 127.0.0.1:50718]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 13:20:30,289-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131829.432, now=131107-132030.288, channel=TCPChannel [type: server, contact: 127.0.0.1:50718]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:20:30,289-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131829.432, now=131107-132030.288, channel=TCPChannel [type: server, contact: 127.0.0.1:50718]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 13:20:30,289-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131829.432, now=131107-132030.288, channel=TCPChannel [type: server, contact: 127.0.0.1:50718]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:20:30,289-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131829.432, now=131107-132030.288, channel=TCPChannel [type: server, contact: 127.0.0.1:50718]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {}
-2013-11-07 13:20:30,289-0600 INFO AbstractCoasterChannel Channel config: null -> {}
-TCPChannel [type: server, contact: 127.0.0.1:50718] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131829.432, now=131107-132030.288, channel=TCPChannel [type: server, contact: 127.0.0.1:50718]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:20:30,289-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50718] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131829.432, now=131107-132030.288, channel=TCPChannel [type: server, contact: 127.0.0.1:50718]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 13:20:30,289-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131829.513, now=131107-132030.376, channel=TCPChannel [type: server, contact: 127.0.0.1:50719]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:20:30,376-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131829.513, now=131107-132030.376, channel=TCPChannel [type: server, contact: 127.0.0.1:50719]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 13:20:30,376-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131829.513, now=131107-132030.376, channel=TCPChannel [type: server, contact: 127.0.0.1:50719]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:20:30,376-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131829.513, now=131107-132030.376, channel=TCPChannel [type: server, contact: 127.0.0.1:50719]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-07 13:20:30,376-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:50719] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131829.513, now=131107-132030.376, channel=TCPChannel [type: server, contact: 127.0.0.1:50719]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:20:30,376-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:50719] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-131829.513, now=131107-132030.376, channel=TCPChannel [type: server, contact: 127.0.0.1:50719]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 13:20:30,376-0600 INFO ChannelManager Channel already shut down
-No streams
-2013-11-07 13:20:31,097-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:20:41,105-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:20:51,113-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:21:01,121-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:21:01,177-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:21:02,750-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:21:11,129-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:21:21,137-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:21:31,145-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 13:21:37,652-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:21:40,068-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:21:41,153-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:21:51,161-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:22:01,169-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:22:11,177-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 13:22:14,126-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 13:22:17,387-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 13:22:21,187-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:22:31,197-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:22:41,216-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:22:50,599-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:22:51,218-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 13:22:54,705-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 13:23:01,221-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:23:11,225-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:23:21,226-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 13:23:27,073-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 13:23:31,235-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:23:32,022-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:23:41,244-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:23:51,256-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:24:01,266-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 13:24:03,548-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:24:09,341-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:24:11,276-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:24:21,289-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:24:31,297-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 13:24:40,021-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 13:24:41,306-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:24:46,658-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:24:51,316-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:25:01,333-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:25:11,351-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:25:16,495-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:25:21,364-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 13:25:23,976-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 13:25:31,376-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:25:41,388-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:25:51,398-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:25:52,969-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:26:01,295-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:26:01,408-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:26:11,418-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:26:21,426-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:26:29,443-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:26:31,435-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 13:26:38,613-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 13:26:41,443-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:26:51,457-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:27:01,465-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 13:27:05,918-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 13:27:11,475-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:27:15,930-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:27:21,486-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:27:31,497-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:27:41,506-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:27:42,391-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:27:51,515-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:27:53,248-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:28:01,528-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:28:11,540-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 13:28:18,865-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 13:28:21,555-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:28:30,566-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:28:31,564-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:28:41,572-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:28:51,579-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:28:55,339-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:29:01,595-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 13:29:07,885-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 13:29:11,604-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:29:21,611-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:29:31,619-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:29:31,813-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:29:41,627-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:29:45,202-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:29:51,635-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:30:01,643-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 13:30:08,287-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 13:30:11,650-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:30:21,658-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:30:22,520-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:30:31,666-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:30:41,674-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 14ms
-2013-11-07 13:30:44,774-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 14ms
-No streams
-2013-11-07 13:30:51,682-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 13:30:59,838-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 13:31:01,689-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:31:11,698-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 13:31:21,236-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 13:31:21,706-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:31:31,714-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:31:37,156-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:31:41,722-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:31:51,730-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:31:57,709-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:32:01,738-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:32:11,746-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 13:32:14,475-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 13:32:21,754-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:32:31,762-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:32:34,183-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:32:41,770-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:32:51,778-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 3ms
-2013-11-07 13:32:51,794-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 3ms
-No streams
-2013-11-07 13:33:01,787-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 13:33:10,657-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 13:33:11,795-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:33:21,803-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:33:29,110-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:33:31,811-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:33:41,819-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:33:47,131-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:33:51,828-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:34:01,836-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:34:06,428-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:34:11,844-0600 INFO Multiplexer No streams
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 13:34:13,950-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:53124] for addition
-2013-11-07 13:34:13,951-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:53124] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:53124]
-2013-11-07 13:34:13,951-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:53124]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f42
-2013-11-07 13:34:13,991-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f42
-MetaChannel [context: cps-53124, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:53124]
-2013-11-07 13:34:13,991-0600 INFO MetaChannel MetaChannel [context: cps-53124, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:53124]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-07 13:34:13,991-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-07 13:34:14,011-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-07 13:34:14,011-0600 INFO Sender Sender TCPChannel queue size: 0
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 13:34:14,052-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:53125] for addition
-2013-11-07 13:34:14,052-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:53125] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:53125]
-2013-11-07 13:34:14,052-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:53125]
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-07 13:34:14,092-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f41
-2013-11-07 13:34:14,093-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f41
-MetaChannel [context: cps-53125, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:53125]
-2013-11-07 13:34:14,093-0600 INFO MetaChannel MetaChannel [context: cps-53125, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:53125]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-07 13:34:14,093-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-07 13:34:14,113-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-07 13:34:14,134-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 129.0) submitting urn:job-0-1383759275014-1383759275015: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 13:34:14,135-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 129.0) submitting urn:job-0-1383759275014-1383759275015: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 13:34:14,135-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-07 13:34:14,135-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-07 13:34:14,142-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7590, JOBSTATUS) unregistering (send)
-2013-11-07 13:34:14,142-0600 INFO RequestHandler Handler(tag: 7590, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 13:34:14,152-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:53126] for addition
-2013-11-07 13:34:14,152-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:53126] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:53126]
-2013-11-07 13:34:14,152-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:53126]
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f40
-2013-11-07 13:34:14,154-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f40
-MetaChannel [context: cps-53126, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:53126]
-2013-11-07 13:34:14,154-0600 INFO MetaChannel MetaChannel [context: cps-53126, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:53126]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-07 13:34:14,154-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-07 13:34:14,194-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-07 13:34:14,195-0600 INFO RequestHandler Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-07 13:34:14,235-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-crush.mcs.anl.gov:0 (quality: 48.0) submitting urn:job-1-1383759275017-1383759275018: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 13:34:14,235-0600 INFO Cpu crush.mcs.anl.gov:0 (quality: 48.0) submitting urn:job-1-1383759275017-1383759275018: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 13:34:14,242-0600 DEBUG Cpu Active workerid=crush.mcs.anl.gov:000000
-Handler(tag: 127, JOBSTATUS) unregistering (send)
-2013-11-07 13:34:14,242-0600 INFO RequestHandler Handler(tag: 127, JOBSTATUS) unregistering (send)
-Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-07 13:34:14,295-0600 INFO RequestHandler Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-07 13:34:16,167-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 13:34:16,167-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:34:16,167-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:34:16,167-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Channel config: null -> {keepalive=null}
-2013-11-07 13:34:16,167-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-07 13:34:16,167-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-TCPChannel [type: server, contact: 127.0.0.1:53124] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:34:16,167-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:53124] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-grind.mcs.anl.gov:0 (quality: 130.0) submitting urn:job-2-1383759275020-1383759275021: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 13:34:16,167-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 130.0) submitting urn:job-2-1383759275020-1383759275021: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-Handler(tag: 7591, JOBSTATUS) unregistering (send)
-2013-11-07 13:34:16,167-0600 INFO RequestHandler Handler(tag: 7591, JOBSTATUS) unregistering (send)
-null: Channel shut down
-2013-11-07 13:34:16,168-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-53124, boundTo: TCPChannel [type: server, contact: 127.0.0.1:53124]] binding to NullChannel
-2013-11-07 13:34:16,168-0600 INFO MetaChannel MetaChannel [context: cps-53124, boundTo: TCPChannel [type: server, contact: 127.0.0.1:53124]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:53124]
-2013-11-07 13:34:16,168-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:53124]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:34:16,168-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 13:34:16,168-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:34:16,168-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 13:34:16,168-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:34:16,168-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 13:34:16,168-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:34:16,168-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 13:34:16,168-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:34:16,168-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 13:34:16,169-0600 INFO ChannelManager Channel exception handled
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:34:16,169-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel config: null -> {}
-2013-11-07 13:34:16,169-0600 INFO AbstractCoasterChannel Channel config: null -> {}
-TCPChannel [type: server, contact: 127.0.0.1:53126] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:34:16,169-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:53126] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-07 13:34:16,169-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-53126, boundTo: TCPChannel [type: server, contact: 127.0.0.1:53126]] binding to NullChannel
-2013-11-07 13:34:16,169-0600 INFO MetaChannel MetaChannel [context: cps-53126, boundTo: TCPChannel [type: server, contact: 127.0.0.1:53126]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:53126]
-2013-11-07 13:34:16,169-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:53126]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:34:16,169-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 13:34:16,169-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:34:16,169-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 13:34:16,169-0600 INFO ChannelManager Channel exception handled
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 13:34:16,171-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 13:34:16,173-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 13:34:16,173-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7592, JOBSTATUS) unregistering (send)
-2013-11-07 13:34:16,174-0600 INFO RequestHandler Handler(tag: 7592, JOBSTATUS) unregistering (send)
-Channel is closed. Removing.
-2013-11-07 13:34:16,181-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 13:34:16,181-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {}
-2013-11-07 13:34:16,181-0600 INFO AbstractCoasterChannel Channel config: null -> {}
-TCPChannel [type: server, contact: 127.0.0.1:53126] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 13:34:16,181-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:53126] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 13:34:16,181-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:53126]
-2013-11-07 13:34:16,181-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:53126]
-Channel is closed. Removing.
-2013-11-07 13:34:16,181-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 13:34:16,181-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-07 13:34:16,181-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:53124] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 13:34:16,181-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:53124] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 13:34:16,181-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:53124]
-2013-11-07 13:34:16,181-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:53124]
-2013-11-07 13:34:16,432-0600 DEBUG Cpu Completed
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:34:16,432-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-crush.mcs.anl.gov:0 jobTerminated
-2013-11-07 13:34:16,432-0600 INFO Cpu crush.mcs.anl.gov:0 jobTerminated
-Channel config: null -> {keepalive=null}
-2013-11-07 13:34:16,432-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-07 13:34:16,433-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-TCPChannel [type: server, contact: 127.0.0.1:53125] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:34:16,433-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:53125] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 128, JOBSTATUS) unregistering (send)
-2013-11-07 13:34:16,433-0600 INFO RequestHandler Handler(tag: 128, JOBSTATUS) unregistering (send)
-2013-11-07 13:34:16,433-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-null: Channel shut down
-2013-11-07 13:34:16,433-0600 INFO AbstractTCPChannel null: Channel shut down
-2013-11-07 13:34:16,433-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-MetaChannel [context: cps-53125, boundTo: TCPChannel [type: server, contact: 127.0.0.1:53125]] binding to NullChannel
-2013-11-07 13:34:16,433-0600 INFO MetaChannel MetaChannel [context: cps-53125, boundTo: TCPChannel [type: server, contact: 127.0.0.1:53125]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:53125]
-2013-11-07 13:34:16,433-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:53125]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:34:16,433-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 13:34:16,433-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:34:16,433-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 13:34:16,433-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:34:16,433-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 13:34:16,433-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:34:16,433-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 13:34:16,434-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:34:16,434-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 13:34:16,434-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-07 13:34:16,442-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 13:34:16,442-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-07 13:34:16,442-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:53125] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 13:34:16,442-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:53125] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 13:34:16,442-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:53125]
-2013-11-07 13:34:16,442-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:53125]
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 13:34:18,212-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 13:34:18,212-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 13:34:18,212-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 13:34:18,212-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-07 13:34:18,212-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7593, JOBSTATUS) unregistering (send)
-2013-11-07 13:34:18,212-0600 INFO RequestHandler Handler(tag: 7593, JOBSTATUS) unregistering (send)
-2013-11-07 13:34:18,212-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-No streams
-2013-11-07 13:34:21,858-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:34:23,605-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:34:31,866-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:34:41,874-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 13:34:43,747-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 13:34:51,883-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:35:00,079-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:35:01,891-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:35:11,899-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:35:21,064-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:35:21,908-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:35:31,916-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 13:35:36,554-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 13:35:41,925-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:35:51,933-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:35:58,382-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:36:01,941-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:36:11,949-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 13:36:13,028-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133414.092, now=131107-133614.951, channel=TCPChannel [type: server, contact: 127.0.0.1:53124]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:36:14,951-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133414.092, now=131107-133614.951, channel=TCPChannel [type: server, contact: 127.0.0.1:53124]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 13:36:14,951-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133414.092, now=131107-133614.951, channel=TCPChannel [type: server, contact: 127.0.0.1:53124]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:36:14,951-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133414.092, now=131107-133614.951, channel=TCPChannel [type: server, contact: 127.0.0.1:53124]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 13:36:14,952-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133414.092, now=131107-133614.951, channel=TCPChannel [type: server, contact: 127.0.0.1:53124]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:36:14,952-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133414.092, now=131107-133614.951, channel=TCPChannel [type: server, contact: 127.0.0.1:53124]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 13:36:14,952-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133414.092, now=131107-133614.951, channel=TCPChannel [type: server, contact: 127.0.0.1:53124]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:36:14,952-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133414.092, now=131107-133614.951, channel=TCPChannel [type: server, contact: 127.0.0.1:53124]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 13:36:14,952-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133414.092, now=131107-133614.951, channel=TCPChannel [type: server, contact: 127.0.0.1:53124]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:36:14,952-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133414.092, now=131107-133614.951, channel=TCPChannel [type: server, contact: 127.0.0.1:53124]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-07 13:36:14,952-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:53124] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133414.092, now=131107-133614.951, channel=TCPChannel [type: server, contact: 127.0.0.1:53124]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:36:14,952-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:53124] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133414.092, now=131107-133614.951, channel=TCPChannel [type: server, contact: 127.0.0.1:53124]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 13:36:14,952-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133414.194, now=131107-133615.052, channel=TCPChannel [type: server, contact: 127.0.0.1:53125]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:36:15,052-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133414.194, now=131107-133615.052, channel=TCPChannel [type: server, contact: 127.0.0.1:53125]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 13:36:15,052-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133414.194, now=131107-133615.052, channel=TCPChannel [type: server, contact: 127.0.0.1:53125]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:36:15,052-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133414.194, now=131107-133615.052, channel=TCPChannel [type: server, contact: 127.0.0.1:53125]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 13:36:15,052-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133414.194, now=131107-133615.052, channel=TCPChannel [type: server, contact: 127.0.0.1:53125]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:36:15,052-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133414.194, now=131107-133615.052, channel=TCPChannel [type: server, contact: 127.0.0.1:53125]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 13:36:15,052-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133414.194, now=131107-133615.052, channel=TCPChannel [type: server, contact: 127.0.0.1:53125]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:36:15,052-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133414.194, now=131107-133615.052, channel=TCPChannel [type: server, contact: 127.0.0.1:53125]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 13:36:15,052-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133414.194, now=131107-133615.052, channel=TCPChannel [type: server, contact: 127.0.0.1:53125]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:36:15,052-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133414.194, now=131107-133615.052, channel=TCPChannel [type: server, contact: 127.0.0.1:53125]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-07 13:36:15,052-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:53125] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133414.194, now=131107-133615.052, channel=TCPChannel [type: server, contact: 127.0.0.1:53125]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:36:15,053-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:53125] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133414.194, now=131107-133615.052, channel=TCPChannel [type: server, contact: 127.0.0.1:53125]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 13:36:15,053-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133414.295, now=131107-133615.152, channel=TCPChannel [type: server, contact: 127.0.0.1:53126]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:36:15,152-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133414.295, now=131107-133615.152, channel=TCPChannel [type: server, contact: 127.0.0.1:53126]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 13:36:15,152-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133414.295, now=131107-133615.152, channel=TCPChannel [type: server, contact: 127.0.0.1:53126]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:36:15,152-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133414.295, now=131107-133615.152, channel=TCPChannel [type: server, contact: 127.0.0.1:53126]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {}
-2013-11-07 13:36:15,152-0600 INFO AbstractCoasterChannel Channel config: null -> {}
-TCPChannel [type: server, contact: 127.0.0.1:53126] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133414.295, now=131107-133615.152, channel=TCPChannel [type: server, contact: 127.0.0.1:53126]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:36:15,152-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:53126] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133414.295, now=131107-133615.152, channel=TCPChannel [type: server, contact: 127.0.0.1:53126]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 13:36:15,152-0600 INFO ChannelManager Channel already shut down
-No streams
-2013-11-07 13:36:21,957-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:36:31,965-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 13:36:35,701-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 13:36:41,973-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:36:49,501-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:36:51,981-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:37:01,989-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:37:11,997-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 4ms
-2013-11-07 13:37:13,023-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 4ms
-No streams
-2013-11-07 13:37:22,006-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 13:37:25,975-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 13:37:32,014-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:37:42,022-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:37:50,338-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:37:52,030-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:38:02,038-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 13:38:02,450-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 13:38:12,046-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:38:22,054-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:38:27,656-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:38:32,062-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:38:38,923-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:38:42,070-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:38:52,078-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:39:02,086-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:39:04,974-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:39:12,094-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:39:15,398-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 13:39:20,915-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:53162] for addition
-2013-11-07 13:39:20,916-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:53162] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:53162]
-2013-11-07 13:39:20,916-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:53162]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f3f
-2013-11-07 13:39:20,940-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f3f
-MetaChannel [context: cps-53162, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:53162]
-2013-11-07 13:39:20,940-0600 INFO MetaChannel MetaChannel [context: cps-53162, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:53162]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-07 13:39:20,940-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-07 13:39:20,981-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-07 13:39:20,981-0600 INFO Sender Sender TCPChannel queue size: 0
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 13:39:21,020-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:53163] for addition
-2013-11-07 13:39:21,020-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:53163] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:53163]
-2013-11-07 13:39:21,020-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:53163]
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f3e
-2013-11-07 13:39:21,021-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f3e
-MetaChannel [context: cps-53163, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:53163]
-2013-11-07 13:39:21,021-0600 INFO MetaChannel MetaChannel [context: cps-53163, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:53163]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-07 13:39:21,022-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-07 13:39:21,062-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-07 13:39:21,062-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-07 13:39:21,099-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 131.0) submitting urn:job-0-1383759275023-1383759275024: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 13:39:21,099-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 131.0) submitting urn:job-0-1383759275023-1383759275024: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 13:39:21,100-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-07 13:39:21,100-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 13:39:21,100-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:53164] for addition
-2013-11-07 13:39:21,100-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:53164] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:53164]
-2013-11-07 13:39:21,100-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:53164]
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f3d
-2013-11-07 13:39:21,103-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f3d
-MetaChannel [context: cps-53164, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:53164]
-2013-11-07 13:39:21,103-0600 INFO MetaChannel MetaChannel [context: cps-53164, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:53164]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-07 13:39:21,103-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-07 13:39:21,107-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7594, JOBSTATUS) unregistering (send)
-2013-11-07 13:39:21,107-0600 INFO RequestHandler Handler(tag: 7594, JOBSTATUS) unregistering (send)
-Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-07 13:39:21,144-0600 INFO RequestHandler Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-07 13:39:21,144-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-07 13:39:21,150-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-crush.mcs.anl.gov:0 (quality: 49.0) submitting urn:job-1-1383759275026-1383759275027: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 13:39:21,150-0600 INFO Cpu crush.mcs.anl.gov:0 (quality: 49.0) submitting urn:job-1-1383759275026-1383759275027: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 13:39:21,156-0600 DEBUG Cpu Active workerid=crush.mcs.anl.gov:000000
-Handler(tag: 129, JOBSTATUS) unregistering (send)
-2013-11-07 13:39:21,156-0600 INFO RequestHandler Handler(tag: 129, JOBSTATUS) unregistering (send)
-Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-07 13:39:21,244-0600 INFO RequestHandler Handler(tag: 42566, SUBMITJOB) unregistering (send)
-Avg stream buf: 0
-2013-11-07 13:39:22,107-0600 INFO Multiplexer Avg stream buf: 0
-2013-11-07 13:39:23,142-0600 DEBUG Cpu Completed
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:39:23,142-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 13:39:23,142-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-Channel config: null -> {keepalive=null}
-2013-11-07 13:39:23,142-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-07 13:39:23,142-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-TCPChannel [type: server, contact: 127.0.0.1:53162] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:39:23,142-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:53162] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 7595, JOBSTATUS) unregistering (send)
-2013-11-07 13:39:23,142-0600 INFO RequestHandler Handler(tag: 7595, JOBSTATUS) unregistering (send)
-2013-11-07 13:39:23,142-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-null: Channel shut down
-2013-11-07 13:39:23,143-0600 INFO AbstractTCPChannel null: Channel shut down
-grind.mcs.anl.gov:0 (quality: 132.0) submitting urn:job-2-1383759275029-1383759275030: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 13:39:23,143-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 132.0) submitting urn:job-2-1383759275029-1383759275030: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-MetaChannel [context: cps-53162, boundTo: TCPChannel [type: server, contact: 127.0.0.1:53162]] binding to NullChannel
-2013-11-07 13:39:23,143-0600 INFO MetaChannel MetaChannel [context: cps-53162, boundTo: TCPChannel [type: server, contact: 127.0.0.1:53162]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:53162]
-2013-11-07 13:39:23,143-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:53162]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:39:23,143-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 13:39:23,143-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:39:23,143-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 13:39:23,143-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:39:23,143-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 13:39:23,144-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:39:23,144-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 13:39:23,144-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:39:23,144-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 13:39:23,144-0600 INFO ChannelManager Channel exception handled
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:39:23,144-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel config: null -> {keepalive=null}
-2013-11-07 13:39:23,144-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:53164] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:39:23,144-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:53164] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-07 13:39:23,144-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-53164, boundTo: TCPChannel [type: server, contact: 127.0.0.1:53164]] binding to NullChannel
-2013-11-07 13:39:23,144-0600 INFO MetaChannel MetaChannel [context: cps-53164, boundTo: TCPChannel [type: server, contact: 127.0.0.1:53164]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:53164]
-2013-11-07 13:39:23,144-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:53164]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:39:23,144-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 13:39:23,144-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:39:23,144-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 13:39:23,144-0600 INFO ChannelManager Channel exception handled
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 13:39:23,147-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 13:39:23,149-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 13:39:23,149-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7596, JOBSTATUS) unregistering (send)
-2013-11-07 13:39:23,149-0600 INFO RequestHandler Handler(tag: 7596, JOBSTATUS) unregistering (send)
-Channel is closed. Removing.
-2013-11-07 13:39:23,150-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 13:39:23,150-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-07 13:39:23,150-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:53164] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 13:39:23,150-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:53164] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 13:39:23,150-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:53164]
-2013-11-07 13:39:23,150-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:53164]
-Channel is closed. Removing.
-2013-11-07 13:39:23,150-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 13:39:23,150-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-07 13:39:23,150-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:53162] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 13:39:23,150-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:53162] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 13:39:23,151-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:53162]
-2013-11-07 13:39:23,151-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:53162]
-2013-11-07 13:39:23,284-0600 DEBUG Cpu Completed
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:39:23,284-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-crush.mcs.anl.gov:0 jobTerminated
-2013-11-07 13:39:23,284-0600 INFO Cpu crush.mcs.anl.gov:0 jobTerminated
-Channel config: null -> {keepalive=null}
-2013-11-07 13:39:23,284-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-07 13:39:23,284-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-TCPChannel [type: server, contact: 127.0.0.1:53163] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:39:23,284-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:53163] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 130, JOBSTATUS) unregistering (send)
-2013-11-07 13:39:23,284-0600 INFO RequestHandler Handler(tag: 130, JOBSTATUS) unregistering (send)
-2013-11-07 13:39:23,284-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-null: Channel shut down
-2013-11-07 13:39:23,284-0600 INFO AbstractTCPChannel null: Channel shut down
-2013-11-07 13:39:23,284-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-MetaChannel [context: cps-53163, boundTo: TCPChannel [type: server, contact: 127.0.0.1:53163]] binding to NullChannel
-2013-11-07 13:39:23,284-0600 INFO MetaChannel MetaChannel [context: cps-53163, boundTo: TCPChannel [type: server, contact: 127.0.0.1:53163]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:53163]
-2013-11-07 13:39:23,285-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:53163]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:39:23,285-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 13:39:23,285-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:39:23,285-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 13:39:23,285-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:39:23,285-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 13:39:23,285-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:39:23,285-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 13:39:23,285-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:39:23,285-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 13:39:23,285-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-07 13:39:23,291-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 13:39:23,291-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-07 13:39:23,291-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:53163] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 13:39:23,291-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:53163] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 13:39:23,291-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:53163]
-2013-11-07 13:39:23,291-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:53163]
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 13:39:25,165-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 13:39:25,165-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 13:39:25,165-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 13:39:25,165-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-07 13:39:25,165-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7597, JOBSTATUS) unregistering (send)
-2013-11-07 13:39:25,165-0600 INFO RequestHandler Handler(tag: 7597, JOBSTATUS) unregistering (send)
-2013-11-07 13:39:25,165-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-No streams
-2013-11-07 13:39:32,116-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:39:42,124-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 13:39:42,293-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:39:51,871-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:39:52,132-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:40:02,140-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:40:12,149-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 13:40:19,611-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 13:40:22,157-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 13:40:28,346-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 13:40:32,165-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:40:42,173-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:40:52,181-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:40:56,928-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:41:02,190-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:41:04,819-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:41:12,198-0600 INFO Multiplexer No streams
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133921.062, now=131107-134121.916, channel=TCPChannel [type: server, contact: 127.0.0.1:53162]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:41:21,916-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133921.062, now=131107-134121.916, channel=TCPChannel [type: server, contact: 127.0.0.1:53162]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 13:41:21,917-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133921.062, now=131107-134121.916, channel=TCPChannel [type: server, contact: 127.0.0.1:53162]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:41:21,917-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133921.062, now=131107-134121.916, channel=TCPChannel [type: server, contact: 127.0.0.1:53162]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 13:41:21,917-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133921.062, now=131107-134121.916, channel=TCPChannel [type: server, contact: 127.0.0.1:53162]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:41:21,917-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133921.062, now=131107-134121.916, channel=TCPChannel [type: server, contact: 127.0.0.1:53162]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 13:41:21,917-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133921.062, now=131107-134121.916, channel=TCPChannel [type: server, contact: 127.0.0.1:53162]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:41:21,917-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133921.062, now=131107-134121.916, channel=TCPChannel [type: server, contact: 127.0.0.1:53162]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 13:41:21,917-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133921.062, now=131107-134121.916, channel=TCPChannel [type: server, contact: 127.0.0.1:53162]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:41:21,917-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133921.062, now=131107-134121.916, channel=TCPChannel [type: server, contact: 127.0.0.1:53162]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-07 13:41:21,917-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:53162] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133921.062, now=131107-134121.916, channel=TCPChannel [type: server, contact: 127.0.0.1:53162]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:41:21,917-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:53162] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133921.062, now=131107-134121.916, channel=TCPChannel [type: server, contact: 127.0.0.1:53162]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 13:41:21,917-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133921.144, now=131107-134122.020, channel=TCPChannel [type: server, contact: 127.0.0.1:53163]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:41:22,020-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133921.144, now=131107-134122.020, channel=TCPChannel [type: server, contact: 127.0.0.1:53163]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 13:41:22,020-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133921.144, now=131107-134122.020, channel=TCPChannel [type: server, contact: 127.0.0.1:53163]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:41:22,020-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133921.144, now=131107-134122.020, channel=TCPChannel [type: server, contact: 127.0.0.1:53163]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 13:41:22,020-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133921.144, now=131107-134122.020, channel=TCPChannel [type: server, contact: 127.0.0.1:53163]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:41:22,020-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133921.144, now=131107-134122.020, channel=TCPChannel [type: server, contact: 127.0.0.1:53163]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 13:41:22,021-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133921.144, now=131107-134122.020, channel=TCPChannel [type: server, contact: 127.0.0.1:53163]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:41:22,021-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133921.144, now=131107-134122.020, channel=TCPChannel [type: server, contact: 127.0.0.1:53163]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 13:41:22,021-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133921.144, now=131107-134122.020, channel=TCPChannel [type: server, contact: 127.0.0.1:53163]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:41:22,021-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133921.144, now=131107-134122.020, channel=TCPChannel [type: server, contact: 127.0.0.1:53163]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-07 13:41:22,021-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:53163] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133921.144, now=131107-134122.020, channel=TCPChannel [type: server, contact: 127.0.0.1:53163]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:41:22,021-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:53163] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133921.144, now=131107-134122.020, channel=TCPChannel [type: server, contact: 127.0.0.1:53163]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 13:41:22,021-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133921.244, now=131107-134122.100, channel=TCPChannel [type: server, contact: 127.0.0.1:53164]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:41:22,100-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133921.244, now=131107-134122.100, channel=TCPChannel [type: server, contact: 127.0.0.1:53164]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 13:41:22,100-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133921.244, now=131107-134122.100, channel=TCPChannel [type: server, contact: 127.0.0.1:53164]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:41:22,100-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133921.244, now=131107-134122.100, channel=TCPChannel [type: server, contact: 127.0.0.1:53164]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-07 13:41:22,100-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:53164] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133921.244, now=131107-134122.100, channel=TCPChannel [type: server, contact: 127.0.0.1:53164]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:41:22,100-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:53164] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-133921.244, now=131107-134122.100, channel=TCPChannel [type: server, contact: 127.0.0.1:53164]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 13:41:22,100-0600 INFO ChannelManager Channel already shut down
-No streams
-2013-11-07 13:41:22,206-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:41:32,213-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 13:41:34,246-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 13:41:41,293-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 13:41:42,221-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:41:52,229-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:42:02,237-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 13:42:11,565-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 13:42:12,245-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:42:17,767-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:42:22,253-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:42:32,261-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:42:42,269-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:42:48,882-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:42:52,277-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 13:42:54,241-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 13:43:02,285-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:43:12,293-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:43:22,302-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:43:26,200-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:43:30,715-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:43:32,310-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:43:42,318-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:43:52,326-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:44:02,334-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 13:44:03,519-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 13:44:07,189-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 13:44:12,343-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:44:22,351-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:44:32,359-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:44:40,836-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:44:42,367-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:44:43,663-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:44:52,375-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:45:02,384-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:45:12,392-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 13:45:18,154-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:45:20,137-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:45:22,400-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:45:32,408-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:45:42,416-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:45:52,425-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 13:45:55,473-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 13:45:56,611-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 13:46:02,433-0600 INFO Multiplexer No streams
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 13:46:07,166-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:53165] for addition
-2013-11-07 13:46:07,167-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:53165] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:53165]
-2013-11-07 13:46:07,167-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:53165]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f3c
-2013-11-07 13:46:07,207-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f3c
-MetaChannel [context: cps-53165, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:53165]
-2013-11-07 13:46:07,207-0600 INFO MetaChannel MetaChannel [context: cps-53165, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:53165]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-07 13:46:07,207-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-07 13:46:07,227-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-07 13:46:07,227-0600 INFO Sender Sender TCPChannel queue size: 0
-Avg stream buf: 0
-2013-11-07 13:46:12,442-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-07 13:46:22,451-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-07 13:46:32,460-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:46:32,791-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:46:33,085-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-07 13:46:42,469-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-07 13:46:52,478-0600 INFO Multiplexer Avg stream buf: 0
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 13:46:55,305-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:53166] for addition
-2013-11-07 13:46:55,305-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:53166] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:53166]
-2013-11-07 13:46:55,305-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:53166]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f3b
-2013-11-07 13:46:55,327-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f3b
-MetaChannel [context: cps-53166, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:53166]
-2013-11-07 13:46:55,327-0600 INFO MetaChannel MetaChannel [context: cps-53166, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:53166]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-07 13:46:55,327-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-07 13:46:55,368-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-07 13:46:55,368-0600 INFO Sender Sender TCPChannel queue size: 0
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 13:46:55,408-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:53167] for addition
-2013-11-07 13:46:55,409-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:53167] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:53167]
-2013-11-07 13:46:55,409-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:53167]
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-07 13:46:55,448-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f3a
-2013-11-07 13:46:55,449-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f3a
-MetaChannel [context: cps-53167, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:53167]
-2013-11-07 13:46:55,449-0600 INFO MetaChannel MetaChannel [context: cps-53167, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:53167]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-07 13:46:55,449-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-07 13:46:55,469-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-07 13:46:55,473-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 133.0) submitting urn:job-0-1383759275032-1383759275033: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 13:46:55,473-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 133.0) submitting urn:job-0-1383759275032-1383759275033: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 13:46:55,473-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-07 13:46:55,473-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-07 13:46:55,480-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7598, JOBSTATUS) unregistering (send)
-2013-11-07 13:46:55,480-0600 INFO RequestHandler Handler(tag: 7598, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 13:46:55,508-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:53168] for addition
-2013-11-07 13:46:55,508-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:53168] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:53168]
-2013-11-07 13:46:55,508-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:53168]
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f39
-2013-11-07 13:46:55,510-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f39
-MetaChannel [context: cps-53168, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:53168]
-2013-11-07 13:46:55,510-0600 INFO MetaChannel MetaChannel [context: cps-53168, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:53168]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-07 13:46:55,510-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-07 13:46:55,550-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-07 13:46:55,571-0600 INFO RequestHandler Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-07 13:46:55,574-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-crush.mcs.anl.gov:0 (quality: 50.0) submitting urn:job-1-1383759275035-1383759275036: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 13:46:55,574-0600 INFO Cpu crush.mcs.anl.gov:0 (quality: 50.0) submitting urn:job-1-1383759275035-1383759275036: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 13:46:55,581-0600 DEBUG Cpu Active workerid=crush.mcs.anl.gov:000000
-Handler(tag: 131, JOBSTATUS) unregistering (send)
-2013-11-07 13:46:55,581-0600 INFO RequestHandler Handler(tag: 131, JOBSTATUS) unregistering (send)
-Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-07 13:46:55,651-0600 INFO RequestHandler Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-07 13:46:57,504-0600 DEBUG Cpu Completed
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:46:57,504-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 13:46:57,505-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-Channel config: null -> {keepalive=null}
-2013-11-07 13:46:57,505-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-07 13:46:57,505-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-TCPChannel [type: server, contact: 127.0.0.1:53166] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:46:57,505-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:53166] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 7599, JOBSTATUS) unregistering (send)
-2013-11-07 13:46:57,505-0600 INFO RequestHandler Handler(tag: 7599, JOBSTATUS) unregistering (send)
-2013-11-07 13:46:57,505-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-null: Channel shut down
-2013-11-07 13:46:57,505-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-53166, boundTo: TCPChannel [type: server, contact: 127.0.0.1:53166]] binding to NullChannel
-2013-11-07 13:46:57,505-0600 INFO MetaChannel MetaChannel [context: cps-53166, boundTo: TCPChannel [type: server, contact: 127.0.0.1:53166]] binding to NullChannel
-grind.mcs.anl.gov:0 (quality: 134.0) submitting urn:job-2-1383759275038-1383759275039: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 13:46:57,505-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 134.0) submitting urn:job-2-1383759275038-1383759275039: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:53166]
-2013-11-07 13:46:57,505-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:53166]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:46:57,505-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 13:46:57,506-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:46:57,506-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 13:46:57,506-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:46:57,506-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 13:46:57,506-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:46:57,506-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 13:46:57,506-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:46:57,506-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 13:46:57,506-0600 INFO ChannelManager Channel exception handled
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:46:57,507-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel config: null -> {keepalive=null}
-2013-11-07 13:46:57,507-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:53168] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:46:57,507-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:53168] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-07 13:46:57,507-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-53168, boundTo: TCPChannel [type: server, contact: 127.0.0.1:53168]] binding to NullChannel
-2013-11-07 13:46:57,507-0600 INFO MetaChannel MetaChannel [context: cps-53168, boundTo: TCPChannel [type: server, contact: 127.0.0.1:53168]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:53168]
-2013-11-07 13:46:57,507-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:53168]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:46:57,507-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 13:46:57,507-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:46:57,507-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 13:46:57,507-0600 INFO ChannelManager Channel exception handled
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 13:46:57,510-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 13:46:57,511-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 13:46:57,511-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7600, JOBSTATUS) unregistering (send)
-2013-11-07 13:46:57,511-0600 INFO RequestHandler Handler(tag: 7600, JOBSTATUS) unregistering (send)
-Channel is closed. Removing.
-2013-11-07 13:46:57,517-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 13:46:57,517-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-07 13:46:57,517-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:53166] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 13:46:57,517-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:53166] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 13:46:57,517-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:53166]
-2013-11-07 13:46:57,517-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:53166]
-Channel is closed. Removing.
-2013-11-07 13:46:57,517-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 13:46:57,517-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-07 13:46:57,517-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:53168] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 13:46:57,517-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:53168] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 13:46:57,517-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:53168]
-2013-11-07 13:46:57,517-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:53168]
-2013-11-07 13:46:57,887-0600 DEBUG Cpu Completed
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:46:57,887-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-crush.mcs.anl.gov:0 jobTerminated
-2013-11-07 13:46:57,887-0600 INFO Cpu crush.mcs.anl.gov:0 jobTerminated
-Channel config: null -> {keepalive=null}
-2013-11-07 13:46:57,887-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-07 13:46:57,887-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-TCPChannel [type: server, contact: 127.0.0.1:53167] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:46:57,887-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:53167] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 132, JOBSTATUS) unregistering (send)
-2013-11-07 13:46:57,887-0600 INFO RequestHandler Handler(tag: 132, JOBSTATUS) unregistering (send)
-2013-11-07 13:46:57,887-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-07 13:46:57,887-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-null: Channel shut down
-2013-11-07 13:46:57,887-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-53167, boundTo: TCPChannel [type: server, contact: 127.0.0.1:53167]] binding to NullChannel
-2013-11-07 13:46:57,888-0600 INFO MetaChannel MetaChannel [context: cps-53167, boundTo: TCPChannel [type: server, contact: 127.0.0.1:53167]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:53167]
-2013-11-07 13:46:57,888-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:53167]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:46:57,888-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 13:46:57,888-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:46:57,888-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 13:46:57,888-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:46:57,888-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 13:46:57,888-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:46:57,888-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 13:46:57,888-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 13:46:57,888-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 13:46:57,888-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-07 13:46:57,898-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 13:46:57,898-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-07 13:46:57,898-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:53167] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 13:46:57,898-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:53167] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 13:46:57,898-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:53167]
-2013-11-07 13:46:57,899-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:53167]
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 13:46:59,539-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 13:46:59,539-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 13:46:59,539-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 13:46:59,540-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-07 13:46:59,540-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7601, JOBSTATUS) unregistering (send)
-2013-11-07 13:46:59,540-0600 INFO RequestHandler Handler(tag: 7601, JOBSTATUS) unregistering (send)
-2013-11-07 13:46:59,540-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-Avg stream buf: 0
-2013-11-07 13:47:02,492-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 13:47:09,559-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 13:47:10,109-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-07 13:47:12,501-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-07 13:47:22,510-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-07 13:47:32,518-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-07 13:47:42,527-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 13:47:46,033-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:47:47,427-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-07 13:47:52,536-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-07 13:48:02,545-0600 INFO Multiplexer Avg stream buf: 0
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-134607.227, now=131107-134808.166, channel=TCPChannel [type: server, contact: 127.0.0.1:53165]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:48:08,166-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-134607.227, now=131107-134808.166, channel=TCPChannel [type: server, contact: 127.0.0.1:53165]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-07 13:48:08,166-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:53165] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-134607.227, now=131107-134808.166, channel=TCPChannel [type: server, contact: 127.0.0.1:53165]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:48:08,166-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:53165] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-134607.227, now=131107-134808.166, channel=TCPChannel [type: server, contact: 127.0.0.1:53165]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-null: Channel shut down
-2013-11-07 13:48:08,167-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-53165, boundTo: TCPChannel [type: server, contact: 127.0.0.1:53165]] binding to NullChannel
-2013-11-07 13:48:08,167-0600 INFO MetaChannel MetaChannel [context: cps-53165, boundTo: TCPChannel [type: server, contact: 127.0.0.1:53165]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:53165]
-2013-11-07 13:48:08,167-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:53165]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-134607.227, now=131107-134808.166, channel=TCPChannel [type: server, contact: 127.0.0.1:53165]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:48:08,167-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-134607.227, now=131107-134808.166, channel=TCPChannel [type: server, contact: 127.0.0.1:53165]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel exception handled
-2013-11-07 13:48:08,167-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-07 13:48:08,181-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 13:48:08,181-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-07 13:48:08,181-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:53165] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 13:48:08,181-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:53165] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 13:48:08,181-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:53165]
-2013-11-07 13:48:08,181-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:53165]
-No streams
-2013-11-07 13:48:12,553-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:48:22,507-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:48:22,561-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:48:24,745-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:48:32,570-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:48:42,578-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:48:52,586-0600 INFO Multiplexer No streams
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-134655.448, now=131107-134856.305, channel=TCPChannel [type: server, contact: 127.0.0.1:53166]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:48:56,305-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-134655.448, now=131107-134856.305, channel=TCPChannel [type: server, contact: 127.0.0.1:53166]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 13:48:56,306-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-134655.448, now=131107-134856.305, channel=TCPChannel [type: server, contact: 127.0.0.1:53166]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:48:56,306-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-134655.448, now=131107-134856.305, channel=TCPChannel [type: server, contact: 127.0.0.1:53166]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 13:48:56,306-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-134655.448, now=131107-134856.305, channel=TCPChannel [type: server, contact: 127.0.0.1:53166]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:48:56,306-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-134655.448, now=131107-134856.305, channel=TCPChannel [type: server, contact: 127.0.0.1:53166]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 13:48:56,306-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-134655.448, now=131107-134856.305, channel=TCPChannel [type: server, contact: 127.0.0.1:53166]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:48:56,306-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-134655.448, now=131107-134856.305, channel=TCPChannel [type: server, contact: 127.0.0.1:53166]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 13:48:56,306-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-134655.448, now=131107-134856.305, channel=TCPChannel [type: server, contact: 127.0.0.1:53166]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:48:56,306-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-134655.448, now=131107-134856.305, channel=TCPChannel [type: server, contact: 127.0.0.1:53166]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-07 13:48:56,306-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:53166] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-134655.448, now=131107-134856.305, channel=TCPChannel [type: server, contact: 127.0.0.1:53166]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:48:56,306-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:53166] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-134655.448, now=131107-134856.305, channel=TCPChannel [type: server, contact: 127.0.0.1:53166]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 13:48:56,306-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-134655.550, now=131107-134856.409, channel=TCPChannel [type: server, contact: 127.0.0.1:53167]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:48:56,409-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-134655.550, now=131107-134856.409, channel=TCPChannel [type: server, contact: 127.0.0.1:53167]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 13:48:56,409-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-134655.550, now=131107-134856.409, channel=TCPChannel [type: server, contact: 127.0.0.1:53167]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:48:56,409-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-134655.550, now=131107-134856.409, channel=TCPChannel [type: server, contact: 127.0.0.1:53167]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 13:48:56,409-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-134655.550, now=131107-134856.409, channel=TCPChannel [type: server, contact: 127.0.0.1:53167]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:48:56,410-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-134655.550, now=131107-134856.409, channel=TCPChannel [type: server, contact: 127.0.0.1:53167]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 13:48:56,410-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-134655.550, now=131107-134856.409, channel=TCPChannel [type: server, contact: 127.0.0.1:53167]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:48:56,410-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-134655.550, now=131107-134856.409, channel=TCPChannel [type: server, contact: 127.0.0.1:53167]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 13:48:56,410-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-134655.550, now=131107-134856.409, channel=TCPChannel [type: server, contact: 127.0.0.1:53167]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:48:56,410-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-134655.550, now=131107-134856.409, channel=TCPChannel [type: server, contact: 127.0.0.1:53167]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-07 13:48:56,410-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:53167] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-134655.550, now=131107-134856.409, channel=TCPChannel [type: server, contact: 127.0.0.1:53167]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:48:56,410-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:53167] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-134655.550, now=131107-134856.409, channel=TCPChannel [type: server, contact: 127.0.0.1:53167]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 13:48:56,410-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-134655.651, now=131107-134856.508, channel=TCPChannel [type: server, contact: 127.0.0.1:53168]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:48:56,508-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-134655.651, now=131107-134856.508, channel=TCPChannel [type: server, contact: 127.0.0.1:53168]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 13:48:56,508-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-134655.651, now=131107-134856.508, channel=TCPChannel [type: server, contact: 127.0.0.1:53168]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:48:56,508-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-134655.651, now=131107-134856.508, channel=TCPChannel [type: server, contact: 127.0.0.1:53168]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-07 13:48:56,508-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:53168] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-134655.651, now=131107-134856.508, channel=TCPChannel [type: server, contact: 127.0.0.1:53168]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 13:48:56,508-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:53168] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-134655.651, now=131107-134856.508, channel=TCPChannel [type: server, contact: 127.0.0.1:53168]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 13:48:56,508-0600 INFO ChannelManager Channel already shut down
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 13:48:58,981-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:49:02,063-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:49:02,594-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:49:12,602-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:49:22,611-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:49:32,619-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 13:49:35,456-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:49:39,381-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:49:42,627-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:49:52,635-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:50:02,643-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 13:50:11,929-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 13:50:12,651-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:50:16,699-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:50:22,659-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:50:32,667-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:50:42,675-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 13:50:48,403-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 13:50:52,683-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 13:50:54,017-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 13:51:02,692-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:51:12,700-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:51:22,708-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 13:51:24,878-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 13:51:31,336-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 13:51:32,716-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:51:42,724-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:51:52,733-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:52:01,352-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:52:02,741-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:52:08,654-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:52:12,749-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:52:22,757-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:52:32,766-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 13:52:37,826-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 13:52:42,774-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 13:52:45,972-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 13:52:52,782-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:53:02,790-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:53:12,798-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:53:14,300-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:53:22,806-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 13:53:23,290-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 13:53:32,814-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:53:42,822-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 13:53:50,774-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 13:53:52,830-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 13:54:00,609-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 13:54:02,838-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:54:12,846-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:54:22,854-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 13:54:27,248-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 13:54:32,863-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:54:37,927-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:54:42,871-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:54:52,879-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:55:02,887-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 13:55:03,722-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 13:55:12,896-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 13:55:15,245-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 13:55:22,904-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:55:32,912-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 13:55:40,196-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 13:55:42,920-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:55:52,563-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:55:52,928-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:56:02,936-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:56:12,945-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 13:56:16,671-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 13:56:22,953-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:56:29,881-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:56:32,961-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:56:42,969-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:56:52,977-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 13:56:53,144-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 13:57:02,985-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 13:57:07,199-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 13:57:12,994-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:57:23,002-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:57:29,618-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:57:33,010-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:57:43,018-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 13:57:44,517-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 13:57:53,026-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:58:03,034-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 13:58:06,092-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 13:58:13,042-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 13:58:21,835-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 13:58:23,050-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:58:33,059-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 13:58:42,566-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 13:58:43,067-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:58:53,075-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 13:58:59,154-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 13:59:03,083-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:59:13,092-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 13:59:19,040-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 13:59:23,100-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:59:33,108-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 13:59:36,472-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 13:59:43,116-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 13:59:53,124-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 13:59:55,515-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 14:00:03,132-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:00:13,140-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:00:13,789-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:00:23,148-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 14:00:31,989-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 14:00:33,157-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:00:43,165-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:00:51,107-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:00:53,173-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:01:03,181-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:01:08,462-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:01:13,189-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:01:23,197-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:01:28,425-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:01:33,205-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:01:43,213-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:01:44,936-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:01:53,222-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:02:03,230-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 14:02:05,744-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 14:02:13,238-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 14:02:21,411-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 14:02:23,246-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:02:33,255-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:02:43,061-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:02:43,263-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:02:53,271-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:02:57,884-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:03:03,280-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:03:13,288-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:03:20,379-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:03:23,296-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:03:33,305-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 14:03:34,357-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 14:03:43,313-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:03:53,321-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 14:03:57,697-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 14:04:03,329-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:04:10,832-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:04:13,337-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:04:23,345-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:04:33,353-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:04:35,015-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:04:43,361-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:04:47,306-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:04:53,369-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:05:03,377-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:05:12,333-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:05:13,385-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:05:23,394-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:05:23,780-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:05:33,402-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:05:43,410-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:05:49,652-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:05:53,418-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:06:00,254-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:06:03,427-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:06:13,435-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:06:23,443-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:06:26,970-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:06:33,451-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-2013-11-07 14:06:36,730-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-No streams
-2013-11-07 14:06:43,460-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:06:53,468-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:07:03,476-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:07:04,288-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 14:07:13,202-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 14:07:13,484-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:07:23,492-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:07:33,500-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 14:07:41,606-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 14:07:43,508-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:07:49,676-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:07:53,516-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:08:03,524-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:08:13,532-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:08:18,924-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:08:23,540-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:08:26,150-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:08:33,547-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:08:43,555-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:08:53,563-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:08:56,242-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:09:02,625-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:09:03,570-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:09:13,578-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:09:23,586-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 14:09:33,561-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 14:09:33,594-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:09:39,098-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:09:43,602-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:09:53,610-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:10:03,618-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:10:10,878-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:10:13,626-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:10:15,573-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:10:23,634-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:10:33,642-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:10:43,650-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:10:48,196-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:10:52,047-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:10:53,658-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:11:03,665-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:11:13,673-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:11:23,681-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:11:25,515-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:11:28,521-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:11:33,689-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:11:43,697-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:11:53,705-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 14:12:02,833-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 14:12:03,712-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 14:12:04,994-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 14:12:13,720-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:12:23,728-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:12:33,736-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:12:40,151-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:12:41,469-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:12:43,743-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:12:53,751-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:13:03,759-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:13:13,766-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:13:17,469-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 14:13:17,944-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 14:13:23,774-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:13:33,782-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:13:43,790-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:13:53,797-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:13:54,417-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:13:54,787-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:14:03,805-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:14:13,813-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:14:23,821-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:14:30,891-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 14:14:32,106-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 14:14:33,828-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:14:43,836-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:14:53,843-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:15:03,851-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:15:07,365-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 14:15:09,424-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 14:15:13,859-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:15:23,867-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:15:33,874-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 14:15:43,840-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 14:15:43,882-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:15:46,741-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:15:53,890-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:16:03,897-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:16:13,905-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:16:20,313-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:16:23,912-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:16:24,060-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:16:33,920-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:16:43,928-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:16:53,936-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:16:56,788-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 14:17:01,378-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 14:17:03,943-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:17:13,951-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:17:23,959-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 14:17:33,262-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 14:17:33,967-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 14:17:38,697-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 14:17:43,975-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:17:53,984-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:18:03,992-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 14:18:09,736-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 14:18:14,001-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:18:16,014-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:18:24,009-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:18:34,019-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:18:44,027-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 14:18:46,211-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:18:53,332-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:18:54,037-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:19:04,046-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:19:14,056-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:19:22,684-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:19:24,067-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:19:30,651-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:19:34,076-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:19:44,087-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:19:54,097-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:19:59,158-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:20:04,107-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:20:07,968-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:20:14,116-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:20:24,125-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:20:34,134-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:20:35,632-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:20:44,143-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:20:45,287-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:20:54,152-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:21:04,162-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 14:21:12,107-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 14:21:14,172-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-07 14:21:22,606-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-No streams
-2013-11-07 14:21:24,181-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:21:34,191-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:21:44,201-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 14:21:48,581-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 14:21:54,211-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 14:21:59,924-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 14:22:04,222-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:22:14,232-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:22:24,242-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 14:22:25,055-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 14:22:34,250-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:22:37,241-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:22:44,258-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:22:54,266-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:23:01,528-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:23:04,275-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:23:14,283-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:23:14,559-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:23:24,292-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:23:34,302-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 14:23:38,002-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 14:23:44,310-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:23:51,877-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:23:54,318-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:24:04,326-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:24:14,334-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:24:14,476-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:24:24,342-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:24:29,195-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:24:34,349-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:24:44,357-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 14:24:50,950-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 14:24:54,365-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:25:04,373-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 14:25:06,514-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 14:25:14,382-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:25:24,389-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:25:27,424-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:25:34,398-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:25:43,831-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:25:44,405-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:25:54,413-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:26:03,899-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:26:04,421-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:26:14,430-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:26:21,149-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:26:24,437-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:26:34,445-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:26:40,372-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:26:44,453-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:26:54,461-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:26:58,467-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:27:04,469-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:27:14,478-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 14:27:16,847-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 14:27:24,485-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:27:34,492-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:27:35,785-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:27:44,500-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:27:53,320-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:27:54,508-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:28:04,516-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 14:28:13,103-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 14:28:14,523-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:28:24,531-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 14:28:29,794-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 14:28:34,539-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:28:44,547-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:28:50,421-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:28:54,555-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:29:04,562-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 14:29:06,269-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 14:29:14,570-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:29:24,578-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 14:29:27,739-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 14:29:34,586-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 14:29:42,742-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 14:29:44,593-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:29:54,601-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:30:04,609-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:30:05,058-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:30:14,616-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:30:19,216-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:30:24,624-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:30:34,632-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:30:42,376-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:30:44,639-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:30:54,647-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 14:30:55,690-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 14:31:04,655-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:31:14,662-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:31:19,694-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:31:24,670-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:31:32,164-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:31:34,678-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:31:44,686-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:31:54,693-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 14:31:57,013-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 14:32:04,701-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 14:32:08,638-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 14:32:14,709-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:32:24,717-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 14:32:34,331-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 14:32:34,725-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:32:44,733-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:32:45,112-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:32:54,740-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:33:04,748-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:33:11,648-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:33:14,756-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 14:33:21,586-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 14:33:24,764-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:33:34,772-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:33:44,780-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:33:48,966-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:33:54,788-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:33:58,060-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:34:04,796-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:34:14,803-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:34:24,811-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:34:26,284-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 14:34:34,534-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 14:34:34,819-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:34:44,827-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:34:54,835-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:35:03,602-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:35:04,843-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:35:11,008-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:35:14,851-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:35:24,859-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:35:34,866-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:35:40,921-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:35:44,874-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 4ms
-2013-11-07 14:35:47,485-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 4ms
-No streams
-2013-11-07 14:35:54,882-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:36:04,890-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:36:14,898-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:36:18,238-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:36:23,956-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:36:24,906-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:36:34,914-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:36:44,921-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:36:54,929-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:36:55,556-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:37:00,430-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:37:04,937-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:37:14,945-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:37:24,953-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:37:32,874-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:37:34,960-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:37:36,904-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:37:44,968-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:37:54,976-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:38:04,984-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:38:10,192-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 14:38:13,378-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 14:38:14,992-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:38:25,000-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:38:35,008-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:38:45,016-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 14:38:47,510-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:38:49,852-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:38:55,023-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:39:05,031-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:39:15,039-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:39:24,828-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:39:25,047-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 14:39:26,326-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 14:39:35,055-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:39:45,063-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:39:55,071-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:40:02,146-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:40:02,800-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:40:05,079-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:40:15,087-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:40:25,095-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:40:35,103-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 14:40:39,275-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-07 14:40:39,464-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-No streams
-2013-11-07 14:40:45,111-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:40:55,119-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:41:05,127-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:41:15,135-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:41:15,748-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:41:16,783-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:41:25,143-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:41:35,151-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:41:45,159-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 14:41:52,222-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:41:54,101-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:41:55,168-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:42:05,176-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:42:15,184-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:42:25,192-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 14:42:28,697-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:42:31,419-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:42:35,200-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:42:45,208-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:42:55,216-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 14:43:05,170-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 14:43:05,223-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:43:08,737-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:43:15,231-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:43:25,239-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:43:35,247-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:43:41,644-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:43:45,256-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 14:43:46,054-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 14:43:55,264-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:44:05,272-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:44:15,280-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 14:44:18,119-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:44:23,373-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:44:25,288-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:44:35,296-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:44:45,304-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 14:44:54,593-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 14:44:55,312-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:45:00,691-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:45:05,320-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:45:15,328-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:45:25,336-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:45:31,066-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:45:35,344-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:45:38,009-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:45:45,353-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:45:55,361-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:46:05,369-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:46:07,540-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:46:15,327-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:46:15,377-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:46:25,385-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:46:35,393-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:46:44,014-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:46:45,402-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:46:52,645-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:46:55,409-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:47:05,417-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:47:15,424-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 14:47:20,488-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 14:47:25,431-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:47:29,963-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:47:35,439-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:47:45,446-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:47:55,453-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:47:56,962-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:48:05,461-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:48:07,281-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:48:15,469-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:48:25,477-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:48:33,436-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:48:35,485-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:48:44,599-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:48:45,492-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:48:55,500-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:49:05,508-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:49:09,910-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:49:15,516-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:49:21,917-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:49:25,524-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:49:35,532-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:49:45,540-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 14:49:46,384-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 14:49:55,547-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:49:59,235-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:50:05,555-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:50:15,563-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:50:22,858-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:50:25,571-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:50:35,579-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:50:36,553-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:50:45,586-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:50:55,594-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 14:50:59,332-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 14:51:05,602-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:51:13,871-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:51:15,610-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:51:25,618-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:51:35,625-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 14:51:35,807-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 14:51:45,633-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:51:51,189-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:51:55,641-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:52:05,649-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:52:12,281-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:52:15,656-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:52:25,664-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:52:28,507-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:52:35,672-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:52:45,680-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:52:48,755-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:52:55,687-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:53:05,695-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:53:05,825-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:53:15,703-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:53:25,229-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:53:25,711-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:53:35,718-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:53:43,143-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:53:45,726-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:53:55,733-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:54:01,703-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:54:05,741-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:54:15,749-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:54:20,461-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:54:25,757-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:54:35,764-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 14:54:38,176-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 14:54:45,772-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:54:55,780-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:54:57,779-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:55:05,787-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 14:55:14,651-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 14:55:15,795-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:55:25,802-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:55:35,097-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:55:35,810-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:55:45,818-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:55:51,125-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:55:55,826-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:56:05,835-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 14:56:12,416-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 14:56:15,843-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:56:25,851-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 14:56:27,599-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 14:56:35,859-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:56:45,867-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:56:49,733-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:56:55,875-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:57:04,073-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:57:05,884-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:57:15,892-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:57:25,900-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:57:27,051-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:57:35,908-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 14:57:40,547-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 14:57:45,916-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:57:55,925-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:58:04,369-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:58:05,933-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:58:15,941-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:58:17,021-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:58:25,950-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:58:35,958-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:58:41,687-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:58:45,966-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 14:58:53,496-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 14:58:55,974-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:59:05,982-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:59:15,991-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 14:59:19,006-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 14:59:25,999-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:59:29,969-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 14:59:36,007-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:59:46,015-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 14:59:56,023-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 14:59:56,324-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:00:06,031-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 15:00:06,444-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 15:00:16,039-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:00:26,047-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:00:33,642-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:00:36,055-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:00:42,917-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:00:46,063-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:00:56,071-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:01:06,079-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-07 15:01:10,960-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-No streams
-2013-11-07 15:01:16,087-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:01:19,391-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:01:26,095-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:01:36,103-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:01:46,111-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:01:48,279-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:01:55,865-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:01:56,119-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:02:06,127-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:02:16,135-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:02:25,597-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:02:26,143-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:02:32,339-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:02:36,151-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:02:46,158-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:02:56,166-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 15:03:02,915-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 15:03:06,174-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 15:03:08,813-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 15:03:16,182-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:03:26,189-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:03:36,197-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:03:40,233-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:03:45,287-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:03:46,205-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:03:56,212-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:04:06,220-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:04:16,228-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:04:17,551-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 15:04:21,761-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 15:04:26,236-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:04:36,244-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:04:46,252-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:04:54,870-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:04:56,260-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 15:04:58,236-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 15:05:06,268-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:05:16,276-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:05:26,284-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:05:32,188-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:05:34,710-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:05:36,292-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:05:46,300-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:05:56,308-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:06:06,316-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:06:09,506-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 15:06:11,184-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 15:06:16,325-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:06:26,333-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:06:36,341-0600 INFO Multiplexer No streams
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 15:06:36,967-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:53424] for addition
-2013-11-07 15:06:36,967-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:53424] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:53424]
-2013-11-07 15:06:36,967-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:53424]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f38
-2013-11-07 15:06:37,004-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f38
-MetaChannel [context: cps-53424, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:53424]
-2013-11-07 15:06:37,004-0600 INFO MetaChannel MetaChannel [context: cps-53424, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:53424]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-07 15:06:37,004-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-07 15:06:37,045-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-07 15:06:37,045-0600 INFO Sender Sender TCPChannel queue size: 0
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 15:06:37,084-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:53425] for addition
-2013-11-07 15:06:37,084-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:53425] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:53425]
-2013-11-07 15:06:37,085-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:53425]
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f37
-2013-11-07 15:06:37,105-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f37
-MetaChannel [context: cps-53425, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:53425]
-2013-11-07 15:06:37,106-0600 INFO MetaChannel MetaChannel [context: cps-53425, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:53425]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-07 15:06:37,106-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-07 15:06:37,126-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-07 15:06:37,167-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-07 15:06:37,167-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 135.0) submitting urn:job-0-1383759275041-1383759275042: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 15:06:37,167-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 135.0) submitting urn:job-0-1383759275041-1383759275042: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 15:06:37,168-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-07 15:06:37,168-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-07 15:06:37,174-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7602, JOBSTATUS) unregistering (send)
-2013-11-07 15:06:37,174-0600 INFO RequestHandler Handler(tag: 7602, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-07 15:06:37,208-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:53426] for addition
-2013-11-07 15:06:37,208-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:53426] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:53426]
-2013-11-07 15:06:37,208-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:53426]
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f36
-2013-11-07 15:06:37,227-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f36
-MetaChannel [context: cps-53426, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:53426]
-2013-11-07 15:06:37,228-0600 INFO MetaChannel MetaChannel [context: cps-53426, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:53426]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-07 15:06:37,228-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-07 15:06:37,248-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-07 15:06:37,268-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-crush.mcs.anl.gov:0 (quality: 51.0) submitting urn:job-1-1383759275044-1383759275045: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 15:06:37,268-0600 INFO Cpu crush.mcs.anl.gov:0 (quality: 51.0) submitting urn:job-1-1383759275044-1383759275045: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 15:06:37,275-0600 DEBUG Cpu Active workerid=crush.mcs.anl.gov:000000
-Handler(tag: 133, JOBSTATUS) unregistering (send)
-2013-11-07 15:06:37,275-0600 INFO RequestHandler Handler(tag: 133, JOBSTATUS) unregistering (send)
-Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-07 15:06:37,289-0600 INFO RequestHandler Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-07 15:06:37,369-0600 INFO RequestHandler Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-07 15:06:39,221-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 15:06:39,221-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 15:06:39,221-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 15:06:39,221-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Channel config: null -> {keepalive=null}
-2013-11-07 15:06:39,221-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:53424] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 15:06:39,221-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:53424] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 15:06:39,222-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-null: Channel shut down
-2013-11-07 15:06:39,222-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-53424, boundTo: TCPChannel [type: server, contact: 127.0.0.1:53424]] binding to NullChannel
-2013-11-07 15:06:39,222-0600 INFO MetaChannel MetaChannel [context: cps-53424, boundTo: TCPChannel [type: server, contact: 127.0.0.1:53424]] binding to NullChannel
-grind.mcs.anl.gov:0 (quality: 136.0) submitting urn:job-2-1383759275047-1383759275048: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-07 15:06:39,222-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 136.0) submitting urn:job-2-1383759275047-1383759275048: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-Handler(tag: 7603, JOBSTATUS) unregistering (send)
-2013-11-07 15:06:39,222-0600 INFO RequestHandler Handler(tag: 7603, JOBSTATUS) unregistering (send)
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:53424]
-2013-11-07 15:06:39,222-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:53424]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 15:06:39,223-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 15:06:39,223-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 15:06:39,223-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 15:06:39,223-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 15:06:39,223-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 15:06:39,223-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 15:06:39,223-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 15:06:39,223-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 15:06:39,223-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 15:06:39,223-0600 INFO ChannelManager Channel exception handled
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 15:06:39,224-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel config: null -> {keepalive=null}
-2013-11-07 15:06:39,224-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:53426] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 15:06:39,224-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:53426] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-07 15:06:39,224-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-53426, boundTo: TCPChannel [type: server, contact: 127.0.0.1:53426]] binding to NullChannel
-2013-11-07 15:06:39,224-0600 INFO MetaChannel MetaChannel [context: cps-53426, boundTo: TCPChannel [type: server, contact: 127.0.0.1:53426]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:53426]
-2013-11-07 15:06:39,224-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:53426]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 15:06:39,224-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 15:06:39,224-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 15:06:39,224-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 15:06:39,224-0600 INFO ChannelManager Channel exception handled
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 15:06:39,227-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 15:06:39,228-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 15:06:39,228-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7604, JOBSTATUS) unregistering (send)
-2013-11-07 15:06:39,228-0600 INFO RequestHandler Handler(tag: 7604, JOBSTATUS) unregistering (send)
-Channel is closed. Removing.
-2013-11-07 15:06:39,235-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 15:06:39,235-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-07 15:06:39,235-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:53426] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 15:06:39,235-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:53426] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 15:06:39,235-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:53426]
-2013-11-07 15:06:39,235-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:53426]
-Channel is closed. Removing.
-2013-11-07 15:06:39,235-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 15:06:39,235-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-07 15:06:39,235-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:53424] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 15:06:39,235-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:53424] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 15:06:39,235-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:53424]
-2013-11-07 15:06:39,235-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:53424]
-2013-11-07 15:06:39,479-0600 DEBUG Cpu Completed
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 15:06:39,479-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-crush.mcs.anl.gov:0 jobTerminated
-2013-11-07 15:06:39,479-0600 INFO Cpu crush.mcs.anl.gov:0 jobTerminated
-Channel config: null -> {keepalive=null}
-2013-11-07 15:06:39,479-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-07 15:06:39,479-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-TCPChannel [type: server, contact: 127.0.0.1:53425] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 15:06:39,479-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:53425] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 134, JOBSTATUS) unregistering (send)
-2013-11-07 15:06:39,480-0600 INFO RequestHandler Handler(tag: 134, JOBSTATUS) unregistering (send)
-2013-11-07 15:06:39,480-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-null: Channel shut down
-2013-11-07 15:06:39,480-0600 INFO AbstractTCPChannel null: Channel shut down
-2013-11-07 15:06:39,480-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-MetaChannel [context: cps-53425, boundTo: TCPChannel [type: server, contact: 127.0.0.1:53425]] binding to NullChannel
-2013-11-07 15:06:39,480-0600 INFO MetaChannel MetaChannel [context: cps-53425, boundTo: TCPChannel [type: server, contact: 127.0.0.1:53425]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:53425]
-2013-11-07 15:06:39,480-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:53425]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 15:06:39,480-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 15:06:39,480-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 15:06:39,480-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 15:06:39,480-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 15:06:39,480-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 15:06:39,480-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 15:06:39,480-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 15:06:39,481-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-07 15:06:39,481-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-07 15:06:39,481-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-07 15:06:39,496-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 15:06:39,496-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-07 15:06:39,496-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:53425] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-07 15:06:39,496-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:53425] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-07 15:06:39,496-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:53425]
-2013-11-07 15:06:39,496-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:53425]
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 15:06:41,241-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-07 15:06:41,241-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 15:06:41,241-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-07 15:06:41,241-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-07 15:06:41,241-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-2013-11-07 15:06:41,241-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-Handler(tag: 7605, JOBSTATUS) unregistering (send)
-2013-11-07 15:06:41,241-0600 INFO RequestHandler Handler(tag: 7605, JOBSTATUS) unregistering (send)
-No streams
-2013-11-07 15:06:46,356-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 15:06:46,825-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 15:06:47,659-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 15:06:56,364-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:07:06,373-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:07:16,381-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:07:24,133-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:07:24,142-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:07:26,391-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:07:36,401-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:07:46,411-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:07:56,419-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:08:00,606-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:08:01,461-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:08:06,429-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:08:16,438-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:08:26,446-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:08:36,452-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:08:37,080-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-150637.126, now=131107-150837.968, channel=TCPChannel [type: server, contact: 127.0.0.1:53424]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 15:08:37,968-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-150637.126, now=131107-150837.968, channel=TCPChannel [type: server, contact: 127.0.0.1:53424]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 15:08:37,968-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-150637.126, now=131107-150837.968, channel=TCPChannel [type: server, contact: 127.0.0.1:53424]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 15:08:37,968-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-150637.126, now=131107-150837.968, channel=TCPChannel [type: server, contact: 127.0.0.1:53424]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 15:08:37,968-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-150637.126, now=131107-150837.968, channel=TCPChannel [type: server, contact: 127.0.0.1:53424]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 15:08:37,968-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-150637.126, now=131107-150837.968, channel=TCPChannel [type: server, contact: 127.0.0.1:53424]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 15:08:37,968-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-150637.126, now=131107-150837.968, channel=TCPChannel [type: server, contact: 127.0.0.1:53424]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 15:08:37,968-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-150637.126, now=131107-150837.968, channel=TCPChannel [type: server, contact: 127.0.0.1:53424]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 15:08:37,968-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-150637.126, now=131107-150837.968, channel=TCPChannel [type: server, contact: 127.0.0.1:53424]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 15:08:37,968-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-150637.126, now=131107-150837.968, channel=TCPChannel [type: server, contact: 127.0.0.1:53424]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-07 15:08:37,968-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:53424] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-150637.126, now=131107-150837.968, channel=TCPChannel [type: server, contact: 127.0.0.1:53424]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 15:08:37,968-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:53424] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-150637.126, now=131107-150837.968, channel=TCPChannel [type: server, contact: 127.0.0.1:53424]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 15:08:37,969-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-150637.248, now=131107-150838.084, channel=TCPChannel [type: server, contact: 127.0.0.1:53425]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 15:08:38,084-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-150637.248, now=131107-150838.084, channel=TCPChannel [type: server, contact: 127.0.0.1:53425]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 15:08:38,084-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-150637.248, now=131107-150838.084, channel=TCPChannel [type: server, contact: 127.0.0.1:53425]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 15:08:38,084-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-150637.248, now=131107-150838.084, channel=TCPChannel [type: server, contact: 127.0.0.1:53425]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-07 15:08:38,084-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-150637.248, now=131107-150838.084, channel=TCPChannel [type: server, contact: 127.0.0.1:53425]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 15:08:38,084-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-150637.248, now=131107-150838.084, channel=TCPChannel [type: server, contact: 127.0.0.1:53425]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-07 15:08:38,084-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-150637.248, now=131107-150838.084, channel=TCPChannel [type: server, contact: 127.0.0.1:53425]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 15:08:38,084-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-150637.248, now=131107-150838.084, channel=TCPChannel [type: server, contact: 127.0.0.1:53425]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-07 15:08:38,084-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-150637.248, now=131107-150838.084, channel=TCPChannel [type: server, contact: 127.0.0.1:53425]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 15:08:38,084-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-150637.248, now=131107-150838.084, channel=TCPChannel [type: server, contact: 127.0.0.1:53425]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-07 15:08:38,084-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:53425] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-150637.248, now=131107-150838.084, channel=TCPChannel [type: server, contact: 127.0.0.1:53425]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 15:08:38,084-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:53425] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-150637.248, now=131107-150838.084, channel=TCPChannel [type: server, contact: 127.0.0.1:53425]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 15:08:38,085-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-150637.369, now=131107-150838.208, channel=TCPChannel [type: server, contact: 127.0.0.1:53426]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 15:08:38,208-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-150637.369, now=131107-150838.208, channel=TCPChannel [type: server, contact: 127.0.0.1:53426]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-07 15:08:38,208-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-150637.369, now=131107-150838.208, channel=TCPChannel [type: server, contact: 127.0.0.1:53426]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 15:08:38,208-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-150637.369, now=131107-150838.208, channel=TCPChannel [type: server, contact: 127.0.0.1:53426]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-07 15:08:38,208-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:53426] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-150637.369, now=131107-150838.208, channel=TCPChannel [type: server, contact: 127.0.0.1:53426]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-07 15:08:38,208-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:53426] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131107-150637.369, now=131107-150838.208, channel=TCPChannel [type: server, contact: 127.0.0.1:53426]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-07 15:08:38,208-0600 INFO ChannelManager Channel already shut down
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:08:38,780-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:08:46,468-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:08:56,477-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:09:06,490-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:09:13,554-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 15:09:16,099-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 15:09:16,505-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:09:26,516-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:09:36,527-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:09:46,536-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:09:50,029-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 15:09:53,417-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 15:09:56,545-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:10:06,554-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:10:16,563-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:10:26,503-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:10:26,571-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:10:30,734-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:10:36,580-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:10:46,589-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:10:56,599-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:11:02,977-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:11:06,608-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:11:08,052-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:11:16,617-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:11:26,627-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:11:36,636-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 15:11:39,452-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:11:45,371-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:11:46,645-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:11:56,655-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:12:06,663-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:12:15,925-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:12:16,671-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:12:22,688-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:12:26,679-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:12:36,686-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:12:46,694-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 15:12:52,399-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 15:12:56,702-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:13:00,006-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:13:06,710-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:13:16,718-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:13:26,726-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:13:28,873-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:13:36,734-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:13:37,324-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:13:46,742-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:13:56,750-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:14:05,347-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:14:06,757-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:14:14,642-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:14:16,765-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:14:26,773-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:14:36,781-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:14:41,821-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:14:46,789-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:14:51,960-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:14:56,797-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:15:06,805-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:15:16,813-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 15:15:18,296-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 15:15:26,821-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:15:29,278-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:15:36,828-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:15:46,836-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:15:54,769-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:15:56,844-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:16:06,596-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:16:06,852-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:16:16,860-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:16:26,868-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:16:31,243-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:16:36,876-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:16:43,914-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:16:46,883-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:16:56,891-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:17:06,899-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:17:07,717-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:17:16,907-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:17:21,232-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:17:26,914-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:17:36,922-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 15:17:44,191-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 15:17:46,930-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:17:56,938-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:17:58,550-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:18:06,946-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:18:16,954-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:18:20,665-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:18:26,961-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:18:35,868-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:18:36,969-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:18:46,977-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:18:56,985-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 15:18:57,139-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 15:19:06,992-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:19:13,186-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:19:17,000-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:19:27,007-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:19:33,613-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:19:37,015-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:19:47,023-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:19:50,504-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:19:57,030-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:20:07,038-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:20:10,087-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:20:17,046-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:20:27,053-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:20:27,822-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:20:37,061-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 15:20:46,562-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 15:20:47,068-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:20:57,076-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:21:05,140-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:21:07,084-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:21:17,091-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 15:21:23,035-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 15:21:27,099-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:21:37,106-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:21:42,458-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:21:47,114-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:21:57,122-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:21:59,510-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:22:07,129-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:22:17,137-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 15:22:19,776-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 15:22:27,145-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 15:22:35,985-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 15:22:37,152-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:22:47,160-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:22:57,094-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:22:57,168-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:23:07,175-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:23:12,458-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:23:17,183-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:23:27,191-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:23:34,412-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:23:37,198-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:23:47,206-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 15:23:48,931-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 15:23:57,214-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:24:07,222-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:24:11,730-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:24:17,229-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:24:25,406-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:24:27,237-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:24:37,245-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:24:47,252-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:24:49,048-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:24:57,260-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:25:01,880-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:25:07,268-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:25:17,276-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:25:26,366-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:25:27,283-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:25:37,291-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 15:25:38,354-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 15:25:47,299-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:25:57,307-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:26:03,684-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:26:07,314-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 15:26:14,827-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 15:26:17,322-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:26:27,330-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:26:37,337-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:26:41,002-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:26:47,345-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:26:51,302-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:26:57,353-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:27:07,361-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:27:17,370-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 15:27:18,321-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 15:27:27,378-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 15:27:27,777-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 15:27:37,387-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:27:47,395-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 15:27:55,639-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 15:27:57,404-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:28:04,250-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:28:07,412-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:28:17,420-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:28:27,428-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:28:32,956-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:28:37,436-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 15:28:40,725-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 15:28:47,444-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:28:57,453-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:29:07,462-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:29:10,274-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 15:29:17,199-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 15:29:17,471-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:29:27,482-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:29:37,491-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:29:47,499-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 15:29:47,593-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 15:29:53,673-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 15:29:57,517-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:30:07,518-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:30:17,522-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:30:24,910-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:30:27,541-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:30:30,146-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:30:37,550-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:30:47,559-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:30:57,571-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:31:02,228-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:31:06,620-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:31:07,583-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:31:17,595-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:31:27,604-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:31:37,614-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 15:31:39,547-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 15:31:43,095-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 15:31:47,627-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:31:57,640-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:32:07,650-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:32:16,864-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:32:17,659-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 15:32:19,569-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 15:32:27,668-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:32:37,676-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:32:47,685-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 15:32:54,183-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:32:56,042-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:32:57,692-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:33:07,700-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:33:17,707-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:33:27,718-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:33:31,500-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 15:33:32,517-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 15:33:37,731-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:33:47,741-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:33:57,750-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:34:07,759-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:34:08,818-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:34:08,990-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:34:17,767-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:34:27,775-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:34:37,783-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 15:34:45,465-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:34:46,136-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:34:47,792-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:34:57,800-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:35:07,809-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:35:17,818-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 15:35:21,939-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 15:35:23,454-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 15:35:27,826-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:35:37,834-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:35:47,842-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:35:57,849-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:35:58,413-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:36:00,772-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:36:07,858-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:36:17,866-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:36:27,876-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 15:36:34,888-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 15:36:37,885-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 15:36:38,091-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 15:36:47,895-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:36:57,905-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:37:07,915-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 15:37:11,362-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:37:15,408-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:37:17,925-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:37:27,934-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:37:37,942-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 15:37:47,835-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 15:37:47,952-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 15:37:52,727-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 15:37:57,961-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:38:07,970-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:38:17,979-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:38:24,309-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:38:27,988-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:38:30,044-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:38:37,997-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:38:48,007-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:38:58,017-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:39:00,783-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 15:39:07,362-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 15:39:08,026-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:39:18,036-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:39:28,047-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:39:37,257-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:39:38,056-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:39:44,680-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:39:48,065-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:39:58,073-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:40:08,084-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 15:40:13,732-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 15:40:18,094-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:40:21,998-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:40:28,104-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:40:38,112-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:40:48,121-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 15:40:50,206-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 15:40:58,130-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 15:40:59,316-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 15:41:08,138-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:41:18,146-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 15:41:26,680-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 15:41:28,155-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:41:36,634-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:41:38,162-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:41:48,170-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:41:58,178-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:42:03,153-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:42:08,186-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 15:42:13,952-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 15:42:18,194-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:42:28,202-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:42:38,210-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 15:42:39,627-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 15:42:48,217-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:42:51,270-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:42:58,225-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:43:08,232-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:43:16,101-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:43:18,240-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:43:28,247-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 15:43:28,588-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 15:43:38,255-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:43:48,263-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 15:43:52,576-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 15:43:58,271-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:44:05,906-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:44:08,279-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:44:18,287-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:44:28,294-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:44:29,049-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:44:38,302-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:44:43,225-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:44:48,310-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:44:58,318-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 15:45:05,524-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 15:45:08,326-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:45:18,334-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:45:20,543-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:45:28,342-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:45:38,350-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 15:45:41,997-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 15:45:48,358-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:45:57,861-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:45:58,366-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:46:08,374-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:46:18,382-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 15:46:18,472-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 15:46:28,391-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 15:46:35,179-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 15:46:38,399-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:46:48,407-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 15:46:54,945-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 15:46:58,415-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:47:08,423-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:47:12,497-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:47:18,431-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:47:28,439-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 15:47:31,420-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 15:47:38,447-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:47:48,456-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:47:49,816-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:47:58,464-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:48:07,893-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:48:08,472-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:48:18,480-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:48:27,134-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:48:28,488-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:48:38,496-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 15:48:44,367-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 15:48:48,504-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:48:58,512-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:49:04,452-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:49:08,520-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:49:18,528-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:49:20,841-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:49:28,536-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:49:38,544-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 15:49:41,770-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 15:49:48,552-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 15:49:57,315-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 15:49:58,560-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:50:08,568-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:50:18,577-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:50:19,088-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:50:28,585-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:50:33,789-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:50:38,593-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:50:48,601-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:50:56,407-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:50:58,609-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:51:08,618-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 15:51:10,264-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 15:51:18,626-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:51:28,634-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 15:51:33,725-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 15:51:38,642-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:51:46,737-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:51:48,651-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:51:58,659-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:52:08,667-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:52:11,043-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:52:18,675-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 15:52:23,211-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 15:52:28,683-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:52:38,691-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 15:52:48,361-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 15:52:48,699-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:52:58,708-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:52:59,685-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:53:08,716-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:53:18,724-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:53:25,679-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:53:28,732-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 15:53:36,159-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 15:53:38,740-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:53:48,748-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:53:58,756-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 15:54:02,997-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 15:54:08,765-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:54:12,633-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:54:18,773-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:54:28,781-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:54:38,789-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:54:40,315-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:54:48,797-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 15:54:49,107-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 15:54:58,805-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:55:08,814-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 15:55:17,634-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 15:55:18,822-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:55:25,581-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:55:28,830-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:55:38,838-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:55:48,846-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 15:55:54,951-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 15:55:58,854-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 15:56:02,055-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 15:56:08,862-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:56:18,871-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:56:28,879-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:56:32,269-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 15:56:38,528-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 15:56:38,887-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:56:48,895-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:56:58,903-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:57:08,911-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 15:57:09,587-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 15:57:15,003-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 15:57:18,919-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:57:28,927-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:57:38,935-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:57:46,905-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:57:48,943-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 15:57:51,477-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 15:57:58,951-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:58:08,959-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:58:18,968-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 15:58:24,223-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:58:27,951-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:58:28,976-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:58:38,984-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:58:48,992-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:58:59,000-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:59:01,541-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 15:59:04,425-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 15:59:09,009-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:59:19,017-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:59:29,024-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:59:38,860-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:59:39,032-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 15:59:40,899-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 15:59:49,040-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 15:59:59,049-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:00:09,057-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 16:00:16,177-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 16:00:17,374-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 16:00:19,065-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:00:29,073-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:00:39,081-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:00:49,089-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:00:53,495-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:00:53,847-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:00:59,097-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:01:09,105-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:01:19,113-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:01:29,122-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 16:01:30,322-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 16:01:30,814-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 16:01:39,130-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:01:49,138-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:01:59,146-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:02:06,795-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:02:08,132-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:02:09,155-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:02:19,163-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:02:29,171-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:02:39,179-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 16:02:43,269-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 16:02:45,451-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 16:02:49,187-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:02:59,195-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:03:09,204-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:03:19,212-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 16:03:19,742-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:03:22,768-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:03:29,220-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:03:39,228-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:03:49,236-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 16:03:56,217-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 16:03:59,244-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:04:00,086-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:04:09,252-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:04:19,261-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:04:29,269-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 16:04:32,691-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 16:04:37,404-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 16:04:39,277-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:04:49,285-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:04:59,293-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:05:09,166-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:05:09,301-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:05:14,722-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:05:19,309-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:05:29,317-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:05:39,326-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 16:05:45,641-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 16:05:49,334-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 16:05:52,040-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 16:05:59,342-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:06:09,350-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:06:19,358-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 16:06:22,114-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:06:29,358-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:06:29,366-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:06:39,374-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:06:49,382-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:06:58,589-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:06:59,390-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:07:06,676-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:07:09,398-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:07:19,406-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:07:29,414-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:07:35,062-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:07:39,422-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 16:07:43,994-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 16:07:49,431-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:07:59,439-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:08:09,447-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 16:08:11,536-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 16:08:19,455-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:08:21,312-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:08:29,464-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:08:39,472-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:08:48,010-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:08:49,480-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 16:08:58,630-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 16:08:59,488-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:09:09,496-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:09:19,504-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 16:09:24,484-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 16:09:29,512-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:09:35,948-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:09:39,520-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:09:49,528-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:09:59,536-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:10:00,958-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:10:09,544-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 16:10:13,267-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 16:10:19,552-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:10:29,560-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 16:10:37,433-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 16:10:39,568-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:10:49,576-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:10:50,584-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:10:59,584-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:11:09,592-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:11:13,906-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:11:19,600-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:11:27,902-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:11:29,608-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:11:39,616-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:11:49,624-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 16:11:50,380-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 16:11:59,633-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:12:05,220-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:12:09,641-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:12:19,649-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 16:12:26,854-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 16:12:29,657-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:12:39,665-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:12:42,538-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:12:49,673-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:12:59,681-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 16:13:03,328-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 16:13:09,689-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:13:19,697-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 16:13:19,856-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 16:13:29,707-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:13:39,715-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:13:39,802-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:13:49,726-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 16:13:57,174-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 16:13:59,736-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:14:09,744-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:14:16,276-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:14:19,752-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:14:29,761-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:14:34,492-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:14:39,769-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:14:49,777-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 16:14:52,750-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 16:14:59,786-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:15:09,794-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 16:15:11,811-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 16:15:19,802-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:15:29,224-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:15:29,810-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:15:39,819-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 16:15:49,128-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 16:15:49,827-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:15:59,835-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:16:05,698-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:16:09,844-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:16:19,852-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:16:26,446-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:16:29,860-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:16:39,868-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:16:42,172-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:16:49,876-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:16:59,884-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 16:17:03,765-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 16:17:09,892-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 16:17:18,646-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 16:17:19,900-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:17:29,908-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:17:39,917-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:17:41,082-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:17:49,925-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 16:17:55,119-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 16:17:59,933-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:18:09,942-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 16:18:18,400-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 16:18:19,950-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:18:29,958-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 16:18:31,594-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 16:18:39,967-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:18:49,975-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:18:55,719-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:18:59,983-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 16:19:08,068-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 16:19:09,991-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:19:20,000-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:19:30,008-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:19:33,037-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:19:40,016-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:19:44,542-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:19:50,024-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:20:00,033-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:20:10,041-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 16:20:10,355-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 16:20:20,049-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:20:21,016-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:20:30,057-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:20:40,065-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:20:47,673-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:20:50,073-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 16:20:57,490-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 16:21:00,081-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:21:10,089-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:21:20,097-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 16:21:24,992-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 16:21:30,105-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 16:21:33,964-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 16:21:40,113-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:21:50,121-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:22:00,130-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:22:02,309-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:22:10,138-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:22:10,438-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:22:20,146-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:22:30,154-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 16:22:39,627-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 16:22:40,163-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 16:22:46,912-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 16:22:50,171-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:23:00,179-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:23:10,188-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:23:16,945-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:23:20,196-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:23:23,386-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:23:30,204-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:23:40,212-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:23:50,220-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:23:54,263-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 16:23:59,860-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 16:24:00,228-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:24:10,236-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:24:20,244-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:24:30,252-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:24:31,581-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 16:24:36,333-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 16:24:40,261-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:24:50,269-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:25:00,277-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:25:08,899-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:25:10,285-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 16:25:12,808-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 16:25:20,293-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:25:30,301-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:25:40,308-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 16:25:46,217-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 16:25:49,283-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 16:25:50,316-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:26:00,324-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:26:10,331-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:26:20,338-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:26:23,535-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:26:25,756-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:26:30,346-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:26:40,353-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:26:50,361-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:27:00,368-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 16:27:00,853-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:27:02,231-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:27:10,375-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:27:20,382-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:27:30,390-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:27:38,171-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:27:38,704-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:27:40,398-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:27:50,406-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:28:00,414-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:28:10,422-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:28:15,178-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 16:28:15,489-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 16:28:20,430-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:28:30,438-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:28:40,445-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:28:50,453-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:28:51,653-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:28:52,807-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:29:00,461-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:29:10,469-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:29:20,477-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:29:28,127-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 16:29:30,125-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 16:29:30,485-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:29:40,492-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:29:50,500-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:30:00,508-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 16:30:04,601-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:30:07,443-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:30:10,516-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:30:20,524-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:30:30,532-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:30:40,540-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:30:41,075-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 16:30:44,761-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 16:30:50,547-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:31:00,555-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:31:10,563-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 16:31:17,549-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 16:31:20,571-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:31:22,079-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:31:30,579-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:31:40,587-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:31:50,595-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:31:54,023-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 16:31:59,397-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 16:32:00,602-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:32:10,610-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:32:20,618-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:32:30,497-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:32:30,626-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:32:36,715-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:32:40,634-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:32:50,641-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:33:00,649-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:33:06,972-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:33:10,657-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 16:33:14,033-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 16:33:20,665-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:33:30,672-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:33:40,680-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 16:33:43,445-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 16:33:50,688-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:33:51,351-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:34:00,696-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:34:10,703-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 16:34:19,919-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 16:34:20,711-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 16:34:28,669-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 16:34:30,719-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:34:40,727-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:34:50,735-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 16:34:56,392-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 16:35:00,742-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:35:05,987-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:35:10,750-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:35:20,758-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:35:30,766-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 16:35:32,868-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 16:35:40,774-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 16:35:43,304-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 16:35:50,781-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:36:00,789-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:36:09,341-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:36:10,797-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:36:20,623-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:36:20,805-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:36:30,813-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:36:40,821-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:36:45,815-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:36:50,829-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 16:36:57,940-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 16:37:00,837-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:37:10,844-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:37:20,852-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:37:22,290-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:37:30,860-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:37:35,259-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:37:40,868-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:37:50,876-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:37:58,764-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:38:00,884-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:38:10,892-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 16:38:12,576-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 16:38:20,899-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:38:30,907-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 16:38:35,238-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 16:38:40,915-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:38:49,895-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:38:50,923-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:39:00,931-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:39:10,939-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:39:11,712-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:39:20,946-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:39:27,213-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:39:30,954-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:39:40,962-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 16:39:48,186-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 16:39:50,970-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:40:00,978-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 16:40:04,531-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 16:40:10,985-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:40:20,993-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:40:24,660-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:40:31,001-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:40:41,009-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:40:41,849-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:40:51,016-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:41:01,024-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 16:41:01,134-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 16:41:11,032-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 16:41:19,167-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 16:41:21,039-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:41:31,047-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 16:41:37,608-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 16:41:41,055-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:41:51,063-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:41:56,486-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:42:01,071-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:42:11,079-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 16:42:14,082-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 16:42:21,086-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:42:31,094-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 16:42:33,803-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 16:42:41,102-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:42:50,556-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:42:51,110-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:43:01,118-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:43:11,122-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:43:11,126-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:43:21,134-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:43:27,030-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:43:31,142-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:43:41,150-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:43:48,440-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:43:51,158-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:44:01,166-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:44:03,504-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:44:11,175-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:44:21,183-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:44:25,758-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:44:31,191-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 16:44:39,978-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 16:44:41,199-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:44:51,207-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:45:01,215-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:45:03,076-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:45:11,223-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 16:45:16,452-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 16:45:21,231-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:45:31,239-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 16:45:40,395-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 16:45:41,247-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:45:51,255-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 16:45:52,927-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 16:46:01,263-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:46:11,271-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:46:17,712-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:46:21,279-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 16:46:29,402-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 16:46:31,287-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:46:41,295-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:46:51,303-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 16:46:55,029-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 16:47:01,311-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 16:47:05,875-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 16:47:11,319-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:47:21,327-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:47:31,335-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:47:32,348-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:47:41,343-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 16:47:42,349-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 16:47:51,351-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:48:01,359-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:48:09,666-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:48:11,367-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 16:48:18,824-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 16:48:21,375-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:48:31,383-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:48:41,391-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 16:48:46,984-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 16:48:51,399-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 16:48:55,297-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 16:49:01,407-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:49:11,415-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:49:21,423-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:49:24,302-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:49:31,431-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:49:31,771-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:49:41,439-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:49:51,447-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:50:01,455-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 16:50:01,620-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 5ms
-2013-11-07 16:50:08,249-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 5ms
-No streams
-2013-11-07 16:50:11,463-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:50:21,471-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:50:31,479-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:50:38,938-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:50:41,487-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:50:44,719-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:50:51,495-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:51:01,504-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:51:11,512-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:51:16,257-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 16:51:21,193-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 16:51:21,520-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:51:31,528-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:51:41,536-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:51:51,544-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:51:53,575-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 16:51:57,666-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 16:52:01,552-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:52:11,560-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:52:21,568-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:52:30,893-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:52:31,576-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 16:52:34,141-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 16:52:41,584-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:52:51,592-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:53:01,600-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:53:08,211-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 16:53:10,615-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 16:53:11,608-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:53:21,616-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:53:31,624-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:53:41,632-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 16:53:45,529-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:53:47,089-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:53:51,640-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:54:01,648-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:54:11,656-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:54:21,664-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:54:22,847-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 16:54:23,563-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 16:54:31,672-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:54:41,680-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:54:51,688-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:55:00,037-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:55:00,165-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:55:01,696-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:55:11,704-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:55:21,712-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:55:31,720-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 16:55:36,511-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 16:55:37,483-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 16:55:41,728-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:55:51,736-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:56:01,744-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:56:11,752-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:56:12,985-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:56:14,801-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:56:21,761-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:56:31,769-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:56:41,777-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 16:56:49,459-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 16:56:51,785-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 16:56:52,119-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 16:57:01,793-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:57:11,801-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:57:21,810-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 16:57:25,935-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:57:29,437-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:57:31,818-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:57:41,826-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:57:51,834-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:58:01,842-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:58:02,408-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:58:06,755-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:58:11,850-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:58:21,858-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:58:31,867-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 16:58:38,882-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 16:58:41,875-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:58:44,073-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:58:51,883-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:59:01,891-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:59:11,899-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:59:15,356-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:59:21,391-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 16:59:21,907-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:59:31,915-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 16:59:41,923-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 16:59:51,830-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 16:59:51,931-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 16:59:58,709-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:00:01,939-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:00:11,948-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:00:21,956-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 17:00:28,304-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 17:00:31,964-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:00:36,027-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:00:41,972-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:00:51,981-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:01:01,989-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:01:04,779-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:01:11,997-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:01:13,345-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:01:22,005-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:01:32,012-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:01:41,252-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:01:42,020-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:01:50,663-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:01:52,028-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:02:02,036-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:02:12,044-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 17:02:17,726-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 17:02:22,052-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:02:27,981-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:02:32,060-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:02:42,068-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:02:52,076-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 17:02:54,200-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 17:03:02,084-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 17:03:05,299-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 17:03:12,092-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:03:22,100-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 17:03:30,674-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 17:03:32,108-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:03:42,116-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:03:42,617-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:03:52,124-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:04:02,132-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:04:07,148-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:04:12,140-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 17:04:19,935-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 17:04:22,148-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:04:32,156-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:04:42,164-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 17:04:43,622-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 17:04:52,171-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:04:57,253-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:05:02,179-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:05:12,187-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:05:20,096-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:05:22,195-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:05:32,203-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:05:34,571-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:05:42,211-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:05:52,219-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 17:05:56,570-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 17:06:02,226-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:06:11,889-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:06:12,234-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:06:22,242-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:06:32,250-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:06:33,044-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:06:42,258-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:06:49,207-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:06:52,266-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:07:02,274-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 17:07:09,518-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 17:07:12,282-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:07:22,290-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 17:07:26,525-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 17:07:32,298-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:07:42,306-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 178ms
-2013-11-07 17:07:46,169-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 178ms
-No streams
-2013-11-07 17:07:52,314-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:08:02,322-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:08:03,843-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:08:12,330-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:08:22,337-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:08:22,466-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:08:32,345-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 17:08:41,160-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 17:08:42,353-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:08:52,361-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:08:58,940-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:09:02,368-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:09:12,376-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:09:18,479-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:09:22,384-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:09:32,392-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 17:09:35,414-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 17:09:42,401-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:09:52,409-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:09:55,797-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:10:02,417-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:10:11,888-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:10:12,425-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:10:22,433-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:10:32,441-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 17:10:33,115-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 17:10:42,449-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 17:10:48,362-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 17:10:52,457-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:11:02,465-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:11:10,433-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:11:12,473-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:11:22,481-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:11:24,836-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:11:32,489-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:11:42,497-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 17:11:47,751-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 17:11:52,505-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 17:12:01,310-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 17:12:02,513-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:12:12,521-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:12:22,530-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:12:25,069-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:12:32,538-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 17:12:37,783-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 17:12:42,546-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:12:52,554-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:13:02,387-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:13:02,562-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:13:12,570-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 17:13:14,258-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 17:13:22,578-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:13:32,586-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:13:39,705-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:13:42,594-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:13:50,732-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:13:52,602-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:14:02,610-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:14:12,618-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:14:17,023-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:14:22,625-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 23ms
-2013-11-07 17:14:27,228-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 23ms
-No streams
-2013-11-07 17:14:32,633-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:14:42,640-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:14:52,648-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 17:14:54,341-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 17:15:02,656-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:15:03,680-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:15:12,663-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:15:22,671-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:15:31,660-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:15:32,678-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 17:15:40,154-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 17:15:42,686-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:15:52,693-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:16:02,700-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:16:08,978-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:16:12,708-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:16:16,628-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:16:22,715-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:16:32,722-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:16:42,730-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 17:16:46,296-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 17:16:52,738-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 17:16:53,102-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 17:17:02,746-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:17:12,754-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:17:22,762-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:17:23,615-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 17:17:29,576-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 17:17:32,770-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:17:42,778-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:17:52,786-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:18:00,933-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:18:02,794-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:18:06,050-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:18:12,802-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:18:22,810-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:18:32,818-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:18:38,252-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:18:42,524-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:18:42,826-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:18:52,835-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:19:02,843-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:19:12,851-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:19:15,570-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:19:18,998-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:19:22,859-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:19:32,868-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:19:42,876-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:19:52,884-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:19:52,888-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 17:19:55,472-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 17:20:02,892-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:20:12,901-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:20:22,909-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:20:30,206-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:20:31,946-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:20:32,917-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:20:42,926-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:20:52,934-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:21:02,942-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:21:07,524-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 17:21:08,420-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 17:21:12,950-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:21:22,958-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:21:32,966-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:21:42,974-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:21:44,842-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:21:44,894-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:21:52,982-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:22:02,990-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:22:12,998-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:22:21,368-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 17:22:22,160-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 17:22:23,006-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:22:33,014-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:22:43,022-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:22:53,029-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 17:22:57,842-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:22:59,478-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:23:03,037-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:23:13,045-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:23:23,053-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:23:33,061-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:23:34,317-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 17:23:36,796-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 17:23:43,069-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:23:53,077-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:24:03,085-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-2013-11-07 17:24:10,793-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-No streams
-2013-11-07 17:24:13,093-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:24:14,114-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:24:23,101-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:24:33,109-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:24:43,117-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:24:47,265-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 17:24:51,432-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 17:24:53,125-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:25:03,133-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:25:13,141-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:25:23,149-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 17:25:23,740-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:25:28,751-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:25:33,156-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:25:43,164-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:25:53,172-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:26:00,213-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:26:03,180-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:26:06,069-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:26:13,188-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:26:23,196-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:26:33,204-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 17:26:36,688-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 17:26:43,212-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:26:43,387-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:26:53,219-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:27:03,227-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:27:13,161-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:27:13,235-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:27:20,705-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:27:23,243-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:27:33,251-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:27:43,259-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 17:27:49,635-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 17:27:53,267-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 17:27:58,023-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 17:28:03,275-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:28:13,283-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:28:23,290-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:28:26,109-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:28:33,298-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:28:35,341-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:28:43,306-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:28:53,314-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 17:29:02,583-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 17:29:03,322-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 17:29:12,658-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 17:29:13,330-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:29:23,338-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:29:33,346-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 17:29:39,056-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 17:29:43,353-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:29:49,977-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:29:53,361-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:30:03,369-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:30:13,377-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 17:30:15,531-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 17:30:23,385-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:30:27,295-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:30:33,393-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:30:43,401-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:30:52,005-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:30:53,409-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:31:03,416-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:31:04,613-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:31:13,424-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:31:23,432-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:31:28,479-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:31:33,440-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:31:41,931-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:31:43,448-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:31:53,456-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:32:03,464-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:32:04,953-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:32:13,472-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 17:32:19,249-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 17:32:23,480-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:32:33,488-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 17:32:41,428-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 17:32:43,495-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:32:53,503-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:32:56,568-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:33:03,511-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:33:13,519-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 17:33:17,901-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 17:33:23,527-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:33:33,535-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 17:33:33,885-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 17:33:43,543-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:33:53,551-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:33:54,375-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:34:03,559-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:34:11,204-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:34:13,568-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:34:23,576-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 17:34:30,850-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 17:34:33,584-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:34:43,592-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:34:48,522-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:34:53,600-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:35:03,609-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:35:07,324-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:35:13,617-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:35:23,625-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 17:35:25,840-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 17:35:33,633-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:35:43,641-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:35:43,798-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:35:53,649-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:36:03,158-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:36:03,656-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:36:13,664-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 17:36:20,272-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 17:36:23,672-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:36:33,680-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 17:36:40,476-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 17:36:43,688-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:36:53,696-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 17:36:56,746-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 17:37:03,703-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:37:13,712-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:37:17,794-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:37:23,720-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 17:37:33,220-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 17:37:33,727-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:37:43,735-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:37:53,744-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:37:55,112-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:38:03,752-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 17:38:09,694-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 17:38:13,759-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:38:23,767-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:38:32,430-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:38:33,775-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:38:43,783-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:38:46,169-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:38:53,791-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:39:03,799-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:39:09,748-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:39:13,807-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:39:22,644-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:39:23,815-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:39:33,823-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:39:43,831-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 17:39:47,066-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 17:39:53,839-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:39:59,118-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:40:03,847-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:40:13,855-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:40:23,863-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:40:24,384-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:40:33,870-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:40:35,592-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:40:43,878-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:40:53,886-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:41:01,702-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:41:03,894-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 17:41:12,066-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 17:41:13,902-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:41:23,910-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:41:33,918-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 17:41:39,020-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 17:41:43,926-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:41:48,540-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:41:53,933-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:42:03,941-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:42:13,949-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:42:16,338-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:42:23,957-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:42:25,014-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:42:33,964-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:42:43,972-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 17:42:53,657-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 17:42:53,980-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:43:01,488-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:43:03,988-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:43:13,996-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:43:24,004-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:43:30,975-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:43:34,012-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 17:43:37,962-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 17:43:44,020-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:43:54,028-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:44:04,036-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:44:08,293-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:44:14,043-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 17:44:14,436-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 17:44:24,051-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:44:34,059-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:44:44,066-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:44:45,611-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:44:50,911-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:44:54,074-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:45:04,082-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:45:14,090-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:45:22,929-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:45:24,098-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:45:27,386-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:45:34,105-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:45:44,113-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:45:54,121-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:46:00,248-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:46:03,860-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:46:04,128-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:46:14,136-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:46:24,144-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:46:34,152-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:46:37,565-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 17:46:40,334-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 17:46:44,159-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:46:54,167-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:47:04,174-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:47:14,182-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 17:47:14,882-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 17:47:16,808-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 17:47:24,190-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:47:34,197-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:47:44,205-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:47:52,201-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 17:47:53,282-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 17:47:54,213-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:48:04,221-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:48:14,229-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:48:24,236-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 17:48:29,519-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:48:29,756-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:48:34,244-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:48:44,252-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:48:54,260-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:49:04,268-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 17:49:06,230-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:49:06,837-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:49:14,275-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:49:24,283-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:49:34,291-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:49:42,704-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:49:44,155-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:49:44,298-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:49:54,306-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:50:04,314-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:50:14,322-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 17:50:19,178-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 17:50:21,473-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 17:50:24,330-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:50:34,339-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:50:44,347-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:50:54,354-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 17:50:55,653-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:50:58,791-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:51:04,362-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:51:14,370-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:51:24,378-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 17:51:32,126-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 17:51:34,386-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 17:51:36,109-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 17:51:44,393-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:51:54,401-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:52:04,409-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 17:52:08,601-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:52:13,427-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:52:14,417-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:52:24,425-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:52:34,432-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:52:44,440-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:52:45,074-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 17:52:50,745-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 17:52:54,448-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:53:04,456-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:53:14,464-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 6ms
-2013-11-07 17:53:21,553-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 6ms
-No streams
-2013-11-07 17:53:24,471-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:53:28,063-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:53:34,479-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:53:44,487-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:53:54,495-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 17:53:58,023-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 17:54:04,503-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:54:05,381-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:54:14,511-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:54:24,519-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:54:34,497-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:54:34,527-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:54:42,699-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:54:44,535-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:54:54,542-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:55:04,550-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 17:55:10,972-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 17:55:14,558-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:55:20,017-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:55:24,566-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:55:34,574-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:55:44,582-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 17:55:47,446-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 17:55:54,590-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:55:57,335-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:56:04,597-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:56:14,605-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 17:56:23,920-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 17:56:24,613-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:56:34,621-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:56:34,653-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:56:44,629-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:56:54,636-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:57:00,393-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:57:04,645-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:57:11,971-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:57:14,653-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:57:24,661-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:57:34,669-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 17:57:36,868-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 17:57:44,678-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 17:57:49,290-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 17:57:54,685-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:58:04,693-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 17:58:13,342-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 17:58:14,701-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:58:24,709-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 17:58:26,608-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 17:58:34,717-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:58:44,725-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 17:58:49,816-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 17:58:54,736-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 17:59:03,926-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 17:59:04,746-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:59:14,756-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:59:24,768-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 17:59:26,291-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 17:59:34,777-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 17:59:41,244-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 17:59:44,788-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 17:59:54,798-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:00:02,764-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:00:04,809-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:00:14,819-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 18:00:18,562-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 18:00:24,829-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:00:34,838-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:00:39,238-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:00:44,849-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:00:54,860-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 18:00:55,879-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 18:01:04,870-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:01:14,880-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:01:15,712-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:01:24,893-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 18:01:33,198-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 18:01:34,895-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:01:44,908-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:01:52,186-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:01:54,916-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:02:04,931-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 18:02:10,516-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 18:02:14,947-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:02:24,962-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-2013-11-07 18:02:28,662-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-No streams
-2013-11-07 18:02:34,972-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:02:44,980-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 18:02:47,834-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 18:02:54,988-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:03:04,997-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 18:03:05,134-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 18:03:15,007-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:03:25,016-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:03:25,152-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:03:35,025-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 18:03:41,609-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 18:03:45,033-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:03:55,041-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 18:04:02,471-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 18:04:05,050-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:04:15,061-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:04:18,082-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:04:25,070-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:04:35,078-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:04:39,788-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:04:45,088-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 18:04:54,557-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 18:04:55,099-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:05:05,108-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:05:15,117-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 18:05:17,106-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 18:05:25,125-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:05:31,030-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:05:35,133-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:05:45,141-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 18:05:54,425-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 18:05:55,150-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:06:05,158-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 18:06:07,505-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 18:06:15,167-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:06:25,175-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 18:06:31,743-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 18:06:35,183-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 18:06:43,979-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 18:06:45,193-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:06:55,202-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:07:05,212-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 18:07:09,061-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 18:07:15,222-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 18:07:20,453-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 18:07:25,238-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:07:35,245-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:07:45,251-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 18:07:46,378-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 18:07:55,252-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:07:56,926-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:08:05,260-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:08:15,270-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 18:08:23,697-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 18:08:25,279-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 18:08:33,401-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 18:08:35,287-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:08:45,296-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:08:55,305-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:09:01,016-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:09:05,314-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 18:09:09,874-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 18:09:15,322-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:09:25,330-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:09:35,339-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:09:38,333-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:09:45,348-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 18:09:46,348-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 18:09:55,356-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:10:05,365-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:10:15,374-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:10:15,651-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 18:10:22,823-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 18:10:25,382-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:10:35,391-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:10:45,399-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 18:10:52,969-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 18:10:55,410-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:10:59,296-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:11:05,419-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:11:15,428-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:11:25,437-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 18:11:30,288-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 18:11:35,446-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:11:35,770-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:11:45,455-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:11:55,465-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:12:05,475-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 18:12:07,605-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:12:12,244-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:12:15,486-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:12:25,497-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:12:35,506-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 18:12:44,923-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 18:12:45,515-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 18:12:48,719-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 18:12:55,524-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:13:05,532-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:13:15,541-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 18:13:22,242-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:13:25,192-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:13:25,549-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:13:35,557-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:13:45,564-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:13:55,577-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 18:13:59,559-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 18:14:01,667-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 18:14:05,586-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:14:15,594-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:14:25,604-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:14:35,615-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 18:14:36,878-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 18:14:38,141-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 18:14:45,631-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:14:55,641-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:15:05,649-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:15:14,195-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:15:14,614-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:15:15,657-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:15:25,666-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:15:35,675-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:15:45,684-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 18:15:51,089-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 18:15:51,513-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 18:15:55,692-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:16:05,700-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:16:15,708-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:16:25,716-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 18:16:27,563-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 18:16:28,832-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 18:16:35,727-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:16:45,735-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:16:55,744-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:17:04,036-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:17:05,752-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:17:06,150-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:17:15,760-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:17:25,768-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:17:35,779-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:17:40,510-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 18:17:43,469-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 18:17:45,788-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:17:55,796-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:18:05,804-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:18:15,812-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:18:16,984-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:18:20,787-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:18:25,820-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:18:35,828-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:18:45,836-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:18:53,458-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:18:55,844-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 18:18:58,105-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 18:19:05,851-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:19:15,860-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:19:25,868-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 18:19:29,933-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:19:35,423-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:19:35,876-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:19:45,885-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:19:55,894-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:20:05,903-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 18:20:06,406-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:20:12,741-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:20:15,913-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:20:25,922-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:20:35,930-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 18:20:42,881-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 18:20:45,939-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 18:20:50,060-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 18:20:55,948-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:21:05,957-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:21:15,965-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:21:19,354-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:21:25,975-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:21:27,377-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:21:35,984-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:21:45,992-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 18:21:55,829-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 18:21:56,001-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 18:22:04,695-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 18:22:06,010-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:22:16,019-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:22:26,027-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:22:32,302-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:22:36,036-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 18:22:42,014-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 18:22:46,045-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:22:56,054-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:23:06,062-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 18:23:08,777-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 18:23:16,071-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 18:23:19,332-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 18:23:26,079-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:23:36,088-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:23:45,250-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:23:46,096-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:23:56,105-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 18:23:56,650-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 18:24:06,115-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:24:16,124-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 18:24:21,725-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 18:24:26,133-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 18:24:33,968-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 18:24:36,142-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:24:46,150-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:24:56,159-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 18:24:58,200-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 18:25:06,168-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 18:25:11,287-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 18:25:16,177-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:25:26,186-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 18:25:34,674-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 18:25:36,194-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:25:46,203-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 18:25:48,606-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 18:25:56,212-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:26:06,221-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:26:11,147-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:26:16,229-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 18:26:25,925-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 18:26:26,238-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:26:36,246-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:26:46,254-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 18:26:47,622-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 18:26:56,263-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:27:03,242-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:27:06,272-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:27:16,280-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:27:24,095-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:27:26,288-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:27:36,297-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 18:27:40,561-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 18:27:46,305-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:27:56,315-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 18:28:00,570-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 18:28:06,325-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:28:16,334-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:28:17,878-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:28:26,342-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:28:36,350-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:28:37,043-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:28:46,359-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 18:28:55,196-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 18:28:56,368-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:29:06,376-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:29:13,517-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:29:16,384-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:29:26,392-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:29:32,514-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:29:36,400-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:29:46,409-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 18:29:49,992-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 18:29:56,417-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:30:06,425-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 18:30:09,832-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 18:30:16,433-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:30:26,441-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:30:26,465-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:30:36,449-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:30:46,457-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 18:30:47,151-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 18:30:56,465-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 18:31:02,940-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 18:31:06,473-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:31:16,481-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:31:24,469-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:31:26,489-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:31:36,498-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:31:39,413-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:31:46,506-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:31:56,514-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 18:32:01,787-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 18:32:06,522-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:32:15,887-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:32:16,530-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:32:26,539-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:32:36,547-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:32:39,105-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:32:46,555-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:32:52,361-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:32:56,563-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:33:06,571-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:33:16,423-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:33:16,579-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:33:26,587-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:33:28,835-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:33:36,595-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:33:46,603-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:33:53,741-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:33:56,611-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:34:05,309-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:34:06,619-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:34:16,627-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:34:26,635-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:34:31,059-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:34:36,643-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:34:41,783-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:34:46,651-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:34:56,659-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:35:06,667-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 18:35:08,377-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 18:35:16,675-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:35:18,257-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:35:26,683-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:35:36,691-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:35:45,695-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:35:46,699-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:35:54,731-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:35:56,707-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:36:06,715-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:36:16,723-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 18:36:23,013-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 18:36:26,731-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 18:36:31,205-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 18:36:36,739-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:36:46,747-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:36:56,755-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:37:00,331-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:37:06,763-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 18:37:07,680-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 18:37:16,771-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:37:26,779-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:37:36,787-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:37:37,649-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:37:44,153-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:37:46,795-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:37:56,803-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:38:06,810-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 18:38:14,968-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 18:38:16,818-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:38:20,627-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:38:26,826-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:38:36,834-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:38:46,842-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:38:52,286-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:38:56,850-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:38:57,101-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:39:06,858-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:39:16,865-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:39:26,873-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 18:39:29,604-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:39:33,575-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:39:36,881-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:39:46,889-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:39:56,897-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:40:06,906-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:40:06,922-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 18:40:10,048-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 18:40:16,913-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:40:26,921-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:40:36,929-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:40:44,240-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:40:46,523-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:40:46,937-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:40:56,945-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:41:06,953-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:41:16,960-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 18:41:21,559-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:41:22,997-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:41:26,968-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:41:36,975-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:41:46,983-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:41:56,990-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:41:58,876-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:41:59,471-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:42:06,998-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:42:17,006-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:42:27,013-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 18:42:35,945-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 18:42:36,194-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 18:42:37,021-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:42:47,028-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:42:57,036-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:43:07,044-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:43:12,420-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:43:13,512-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:43:17,051-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:43:27,059-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:43:37,067-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:43:47,075-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 18:43:48,893-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:43:50,830-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:43:57,083-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:44:07,091-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:44:17,098-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:44:25,368-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:44:27,106-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 18:44:28,148-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 18:44:37,114-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:44:47,122-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:44:57,130-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 18:45:01,843-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:45:05,466-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:45:07,138-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:45:17,146-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:45:27,154-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:45:37,162-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:45:38,317-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 18:45:42,784-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 18:45:47,170-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:45:57,178-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:46:07,186-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 18:46:14,790-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 18:46:17,194-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:46:20,102-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:46:27,202-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:46:37,209-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:46:47,217-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:46:51,265-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:46:57,226-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:46:57,420-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:47:07,234-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:47:17,242-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:47:27,250-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 18:47:27,740-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 18:47:34,739-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 18:47:37,258-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:47:47,266-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:47:57,274-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:48:04,213-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:48:07,282-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:48:12,057-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:48:17,290-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:48:27,299-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:48:37,307-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 18:48:40,686-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 18:48:47,315-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 18:48:49,375-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 18:48:57,323-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:49:07,331-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:49:17,161-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:49:17,340-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:49:26,693-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:49:27,348-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:49:37,356-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:49:47,364-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 18:49:53,635-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 18:49:57,372-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:50:04,011-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:50:07,380-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:50:17,389-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:50:27,397-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:50:30,109-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:50:37,405-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 18:50:41,329-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 18:50:47,413-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:50:57,421-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 18:51:06,583-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 18:51:07,429-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:51:17,437-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:51:18,647-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:51:27,445-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:51:37,453-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:51:43,057-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:51:47,462-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 18:51:55,965-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 18:51:57,470-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:52:07,478-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:52:17,486-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 18:52:19,531-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 18:52:27,494-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:52:33,283-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:52:37,502-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:52:47,510-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:52:56,006-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:52:57,518-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:53:07,526-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:53:10,601-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:53:17,536-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:53:27,544-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:53:32,480-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:53:37,552-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:53:47,560-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:53:47,919-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:53:57,568-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:54:07,576-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 18:54:08,955-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 18:54:17,584-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:54:25,237-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:54:27,592-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:54:37,601-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 18:54:45,428-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 18:54:47,609-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:54:57,617-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 18:55:02,555-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 18:55:07,625-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:55:17,633-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:55:21,902-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:55:27,641-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:55:37,649-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:55:39,873-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:55:47,657-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:55:57,666-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 18:55:58,376-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 18:56:07,674-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:56:17,191-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:56:17,682-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:56:27,690-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:56:34,851-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:56:37,698-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:56:47,706-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:56:54,509-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:56:57,714-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:57:07,721-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 18:57:11,325-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 18:57:17,729-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:57:27,737-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:57:31,827-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:57:37,745-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:57:47,753-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:57:47,799-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:57:57,761-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:58:07,769-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:58:09,145-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:58:17,777-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:58:24,273-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:58:27,785-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:58:37,793-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:58:46,463-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:58:47,802-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:58:57,810-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:59:00,747-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:59:07,818-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:59:17,827-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 18:59:23,781-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 18:59:27,835-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 18:59:37,220-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 18:59:37,843-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:59:47,852-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 18:59:57,860-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:00:01,099-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:00:07,868-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:00:13,695-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:00:17,876-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:00:27,884-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:00:37,892-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:00:38,417-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:00:47,900-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 19:00:50,170-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 19:00:57,908-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:01:07,916-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:01:15,735-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:01:17,924-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:01:26,643-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:01:27,932-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:01:37,940-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:01:47,948-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:01:53,053-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:01:57,956-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 19:02:03,116-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 19:02:07,964-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:02:17,972-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:02:27,980-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:02:30,371-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:02:37,988-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:02:39,591-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:02:47,996-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:02:58,004-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:03:07,689-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:03:08,012-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:03:16,065-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:03:18,020-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:03:28,027-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:03:38,035-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:03:45,007-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:03:48,043-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:03:52,539-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:03:58,051-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:04:08,059-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:04:18,067-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:04:22,325-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:04:28,075-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:04:29,013-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:04:38,083-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:04:48,091-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:04:58,099-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:04:59,643-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:05:05,487-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:05:08,107-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:05:18,115-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:05:28,123-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:05:36,961-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:05:38,131-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:05:41,961-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:05:48,139-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:05:58,147-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:06:08,155-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:06:14,279-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:06:18,163-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:06:18,435-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:06:28,170-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:06:38,178-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:06:48,186-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:06:51,597-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:06:54,909-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:06:58,194-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:07:08,202-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:07:18,210-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:07:28,218-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:07:28,915-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:07:31,383-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:07:38,226-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:07:48,233-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:07:58,241-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:08:06,233-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:08:07,857-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:08:08,249-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:08:18,256-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:08:28,264-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:08:38,272-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:08:43,551-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:08:44,331-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:08:48,279-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:08:58,287-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:09:08,294-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:09:18,303-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 19:09:20,806-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 19:09:20,868-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 19:09:28,311-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:09:38,319-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:09:48,327-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:09:57,279-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:09:58,187-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:09:58,335-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:10:08,343-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:10:18,351-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:10:28,359-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 19:10:33,754-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:10:35,505-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:10:38,367-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:10:48,375-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:10:58,383-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:11:08,390-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:11:10,227-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:11:12,823-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:11:18,398-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:11:28,406-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:11:38,414-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:11:46,701-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:11:48,422-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:11:50,141-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:11:58,430-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:12:08,438-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:12:18,446-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:12:23,175-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:12:27,459-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:12:28,454-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:12:38,462-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:12:48,470-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:12:58,478-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:12:59,649-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:13:04,777-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:13:08,486-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:13:18,494-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:13:28,502-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:13:36,123-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:13:38,510-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:13:42,095-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:13:48,518-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:13:58,526-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:14:08,534-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:14:12,597-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:14:18,542-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:14:19,413-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:14:28,550-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:14:38,558-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:14:48,566-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:14:49,071-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:14:56,731-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:14:58,574-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:15:08,582-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:15:18,590-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 19:15:25,546-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 19:15:28,597-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:15:34,049-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:15:38,605-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:15:48,613-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:15:58,621-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:16:02,019-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:16:08,629-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:16:11,367-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:16:18,637-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:16:28,645-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:16:38,494-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:16:38,653-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:16:48,661-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:16:48,685-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:16:58,669-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:17:08,677-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:17:14,968-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:17:18,684-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:17:26,003-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:17:28,692-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:17:38,700-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:17:48,708-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:17:51,443-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:17:58,716-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:18:03,321-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:18:08,724-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:18:18,732-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:18:27,917-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:18:28,740-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:18:38,747-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 19:18:40,639-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 19:18:48,756-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:18:58,764-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:19:04,391-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:19:08,772-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:19:17,958-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:19:18,779-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:19:28,787-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:19:38,795-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:19:40,865-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:19:48,803-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:19:55,276-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:19:58,811-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:20:08,819-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:20:17,339-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:20:18,827-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:20:28,835-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:20:32,594-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:20:38,843-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:20:48,851-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:20:53,813-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:20:58,859-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:21:08,868-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:21:09,912-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:21:18,876-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:21:28,884-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:21:30,287-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:21:38,891-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 19:21:47,230-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 19:21:48,899-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:21:58,906-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:22:06,761-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:22:08,914-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:22:18,921-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:22:24,549-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:22:28,928-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:22:38,936-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 19:22:43,236-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 19:22:48,943-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:22:58,951-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:23:01,867-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:23:08,959-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:23:18,967-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:23:19,709-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:23:28,975-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:23:38,982-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:23:39,185-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:23:48,990-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 19:23:56,184-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 19:23:58,998-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:24:09,006-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:24:16,503-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:24:19,013-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:24:29,021-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:24:32,657-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:24:39,029-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:24:49,037-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:24:53,821-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:24:59,044-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:25:09,052-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:25:09,131-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:25:19,060-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:25:29,068-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:25:31,139-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:25:39,076-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 3ms
-2013-11-07 19:25:45,607-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 3ms
-No streams
-2013-11-07 19:25:49,084-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:25:59,092-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 19:26:08,456-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 19:26:09,100-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:26:19,108-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:26:22,079-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:26:29,116-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:26:39,124-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:26:45,775-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:26:49,132-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 19:26:58,552-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 19:26:59,140-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:27:09,148-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:27:19,156-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:27:23,093-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:27:29,163-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:27:35,027-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:27:39,171-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:27:49,179-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:27:59,187-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:28:00,411-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:28:09,195-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:28:11,501-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:28:19,202-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:28:29,210-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:28:37,729-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:28:39,218-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 19:28:47,976-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 19:28:49,226-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:28:59,234-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:29:09,242-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 19:29:15,046-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 19:29:19,250-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:29:24,449-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:29:29,258-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:29:39,266-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:29:49,273-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:29:52,365-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:29:59,281-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:30:00,923-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:30:09,289-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:30:19,297-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:30:29,305-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:30:29,683-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:30:37,397-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:30:39,313-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:30:49,321-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:30:59,329-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:31:07,002-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:31:09,337-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 19:31:13,872-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 19:31:19,345-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:31:29,353-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:31:39,361-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:31:44,320-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:31:49,369-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:31:50,345-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:31:59,377-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:32:09,385-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:32:19,393-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:32:21,638-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:32:26,819-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:32:29,401-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:32:39,408-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:32:49,416-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:32:58,956-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:32:59,424-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:33:03,293-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:33:09,432-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:33:19,440-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:33:29,448-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:33:36,274-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:33:39,456-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:33:39,767-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:33:49,465-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:33:59,472-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:34:09,481-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:34:13,592-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:34:16,241-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:34:19,489-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:34:29,497-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:34:39,505-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:34:49,513-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:34:50,910-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:34:52,715-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:34:59,521-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:35:09,530-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:35:19,538-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:35:28,228-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:35:29,189-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:35:29,546-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:35:39,554-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:35:49,563-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:35:59,571-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:36:05,546-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:36:05,663-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:36:09,579-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:36:19,587-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:36:29,595-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:36:39,604-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 19:36:42,138-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:36:42,864-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:36:49,612-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:36:59,620-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:37:09,628-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:37:18,611-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:37:19,637-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:37:20,182-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:37:29,645-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:37:39,653-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:37:49,661-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:37:55,085-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:37:57,500-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:37:59,670-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:38:09,678-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:38:19,686-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:38:29,695-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:38:31,559-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:38:34,818-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:38:39,703-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:38:49,711-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:38:59,719-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 19:39:08,034-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 19:39:09,727-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:39:12,136-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:39:19,735-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:39:29,743-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:39:39,752-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:39:44,507-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:39:49,454-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:39:49,760-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:39:59,768-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:40:09,777-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:40:19,785-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:40:20,981-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:40:26,772-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:40:29,793-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:40:39,801-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:40:49,809-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:40:57,455-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:40:59,817-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:41:04,090-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:41:09,825-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:41:19,833-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:41:29,841-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:41:33,929-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:41:39,848-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:41:41,408-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:41:49,856-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:41:59,864-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:42:09,872-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:42:10,403-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:42:18,726-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:42:19,879-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:42:29,887-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:42:39,895-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:42:46,877-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:42:49,903-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:42:56,044-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:42:59,911-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:43:09,918-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:43:19,926-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:43:23,351-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:43:29,934-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:43:33,362-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:43:39,942-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:43:49,950-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:43:59,825-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:43:59,958-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:44:09,965-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:44:10,680-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:44:19,973-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:44:29,981-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:44:36,300-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:44:39,989-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:44:47,998-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:44:49,997-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:45:00,004-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:45:10,012-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:45:12,774-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:45:20,020-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:45:25,316-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:45:30,027-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:45:40,035-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:45:49,248-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:45:50,043-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:46:00,051-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:46:02,634-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:46:10,059-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:46:20,066-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:46:25,722-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:46:30,074-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:46:39,952-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:46:40,082-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:46:50,089-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:47:00,097-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:47:02,197-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:47:10,105-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:47:17,270-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:47:20,113-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:47:30,120-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:47:38,671-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:47:40,128-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:47:50,136-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:47:54,588-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:48:00,144-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:48:10,152-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 19:48:15,145-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 19:48:20,159-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:48:30,167-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 19:48:31,905-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 19:48:40,175-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:48:50,183-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:48:51,620-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:49:00,191-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:49:09,224-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:49:10,198-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:49:20,206-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 19:49:28,094-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 19:49:30,214-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:49:40,222-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:49:46,542-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:49:50,230-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:50:00,238-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:50:04,568-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:50:10,245-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:50:20,253-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:50:23,860-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:50:30,261-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:50:40,269-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 19:50:41,043-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 19:50:50,277-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:51:00,285-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:51:01,178-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:51:10,292-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:51:17,516-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:51:20,300-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:51:30,308-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:51:38,496-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:51:40,316-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:51:50,324-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 19:51:53,990-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 19:52:00,332-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:52:10,339-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:52:15,814-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:52:20,347-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:52:30,355-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:52:30,465-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:52:40,363-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:52:50,371-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:52:53,132-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:53:00,378-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:53:06,939-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:53:10,386-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:53:20,394-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:53:30,402-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:53:30,450-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:53:40,409-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:53:43,413-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:53:50,417-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:54:00,425-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:54:07,768-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:54:10,432-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 19:54:19,886-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 19:54:20,440-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:54:30,448-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:54:40,456-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:54:45,086-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:54:50,463-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:54:56,361-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:55:00,471-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:55:10,479-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:55:20,487-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:55:22,404-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:55:30,494-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:55:32,835-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:55:40,502-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:55:50,510-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:55:59,722-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:56:00,518-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:56:09,309-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:56:10,526-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:56:20,534-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:56:30,542-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:56:37,040-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:56:40,551-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 19:56:45,782-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 19:56:50,558-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:57:00,566-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:57:10,574-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:57:14,358-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:57:20,582-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:57:22,257-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:57:30,591-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:57:40,599-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:57:50,607-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-07 19:57:51,675-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 19:57:58,731-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 19:58:00,615-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:58:10,623-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:58:20,631-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:58:28,994-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:58:30,640-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:58:35,206-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:58:40,648-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:58:50,656-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:59:00,664-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:59:06,312-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:59:10,673-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 3ms
-2013-11-07 19:59:11,682-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 3ms
-No streams
-2013-11-07 19:59:20,681-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:59:30,689-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 19:59:40,698-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:59:43,630-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 19:59:48,154-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 19:59:50,706-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:00:00,714-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:00:10,722-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:00:20,730-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:00:20,948-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 20:00:24,629-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 20:00:30,738-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:00:40,747-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:00:50,755-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-07 20:00:58,267-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-07 20:01:00,763-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:01:01,102-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:01:10,771-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:01:20,780-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:01:30,787-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:01:35,584-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-2013-11-07 20:01:37,578-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-No streams
-2013-11-07 20:01:40,795-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:01:50,803-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:02:00,811-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:02:10,819-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:02:12,902-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:02:14,050-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:02:20,827-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:02:30,834-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:02:40,842-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:02:50,220-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:02:50,524-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:02:50,850-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:03:00,858-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:03:10,865-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:03:20,873-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:03:26,998-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:03:27,538-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:03:30,881-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:03:40,889-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:03:50,897-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:04:00,904-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:04:03,472-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:04:04,856-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:04:10,912-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:04:20,920-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:04:30,927-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 20:04:39,946-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 20:04:40,935-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:04:42,174-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:04:50,943-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:05:00,951-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:05:10,959-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:05:16,421-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-07 20:05:19,491-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-No streams
-2013-11-07 20:05:20,967-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:05:30,974-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:05:40,983-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:05:50,991-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:05:52,895-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:05:56,810-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:06:00,999-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:06:11,007-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:06:21,015-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-2013-11-07 20:06:29,371-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-No streams
-2013-11-07 20:06:31,023-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:06:34,128-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:06:41,031-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:06:51,039-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:07:01,048-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 20:07:05,843-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 20:07:11,056-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:07:11,446-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:07:21,064-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:07:31,072-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:07:41,080-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:07:42,318-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:07:48,764-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:07:51,089-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:08:01,097-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:08:11,105-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:08:18,793-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:08:21,113-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-07 20:08:26,082-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-No streams
-2013-11-07 20:08:31,121-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:08:41,129-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:08:51,137-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:08:55,267-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:09:01,145-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:09:03,401-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:09:11,153-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:09:21,161-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:09:31,169-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 20:09:31,741-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:09:40,719-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:09:41,177-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:09:51,185-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:10:01,193-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:10:08,216-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:10:11,201-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:10:18,037-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:10:21,209-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:10:31,217-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:10:41,225-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:10:44,690-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:10:51,233-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:10:55,355-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:11:01,240-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:11:11,248-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:11:21,164-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:11:21,255-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:11:31,263-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-07 20:11:32,672-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-No streams
-2013-11-07 20:11:41,270-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:11:51,277-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:11:57,638-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:12:01,285-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:12:09,991-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:12:11,292-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:12:21,300-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:12:31,307-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:12:34,112-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:12:41,315-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:12:47,309-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:12:51,323-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:13:01,331-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:13:10,586-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:13:11,338-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:13:21,346-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:13:24,627-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:13:31,354-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:13:41,362-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:13:47,060-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:13:51,369-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:14:01,377-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:14:01,945-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:14:11,385-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:14:21,393-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 20:14:23,534-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 20:14:31,401-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:14:39,263-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:14:41,408-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:14:51,416-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:15:00,008-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:15:01,424-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:15:11,432-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:15:16,581-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:15:21,439-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:15:31,447-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 20:15:36,483-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 20:15:41,455-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:15:51,463-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-07 20:15:53,899-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-07 20:16:01,470-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:16:11,478-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:16:12,956-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:16:21,486-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:16:31,217-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:16:31,494-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:16:41,502-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 20:16:49,430-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 20:16:51,510-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:17:01,517-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:17:08,535-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:17:11,525-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:17:21,533-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:17:25,904-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:17:31,541-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:17:41,548-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:17:45,853-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:17:51,556-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:18:01,564-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 20:18:02,378-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 20:18:11,572-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:18:21,580-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:18:23,171-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:18:31,588-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:18:38,852-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:18:41,595-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:18:51,603-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-07 20:19:00,489-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-No streams
-2013-11-07 20:19:01,611-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:19:11,619-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 20:19:15,326-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 20:19:21,627-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:19:31,635-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:19:37,808-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:19:41,643-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:19:51,651-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:19:51,800-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:20:01,659-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:20:11,667-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:20:15,126-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:20:21,675-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 20:20:28,275-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 20:20:31,683-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:20:41,691-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:20:51,699-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:20:52,444-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:21:01,707-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:21:04,748-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:21:11,715-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:21:21,723-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:21:29,762-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:21:31,731-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 20:21:41,222-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 20:21:41,740-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:21:51,748-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:22:01,756-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:22:07,081-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:22:11,764-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:22:17,696-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:22:21,772-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:22:31,780-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:22:41,788-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:22:44,400-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:22:51,796-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 20:22:54,170-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 20:23:01,804-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:23:11,812-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:23:21,718-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:23:21,820-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:23:30,645-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:23:31,828-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:23:41,836-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:23:51,844-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:23:59,037-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:24:01,852-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:24:07,119-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:24:11,859-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:24:21,867-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:24:31,875-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-07 20:24:36,355-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-No streams
-2013-11-07 20:24:41,883-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 20:24:43,594-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 20:24:51,891-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:25:01,899-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:25:11,907-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:25:13,673-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:25:20,068-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:25:21,915-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:25:31,924-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:25:41,931-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:25:50,991-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:25:51,939-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:25:56,542-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:26:01,947-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:26:11,955-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:26:21,963-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:26:28,309-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:26:31,970-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:26:33,017-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:26:41,978-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:26:51,986-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:27:01,995-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:27:05,627-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 20:27:09,491-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 20:27:12,001-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:27:22,009-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:27:32,018-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:27:42,026-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:27:42,946-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 20:27:45,966-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 20:27:52,034-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:28:02,042-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:28:12,051-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:28:20,264-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:28:22,058-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:28:22,439-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:28:32,066-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:28:42,074-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:28:52,082-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:28:57,581-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 20:28:58,914-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 20:29:02,090-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:29:12,098-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:29:22,106-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:29:32,113-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:29:34,899-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:29:35,387-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:29:42,121-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:29:52,129-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:30:02,137-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 20:30:11,861-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 20:30:12,145-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:30:12,217-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:30:22,152-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:30:32,160-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:30:42,168-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 20:30:48,336-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:30:49,535-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:30:52,176-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:31:02,184-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:31:12,192-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:31:22,199-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 20:31:24,809-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:31:26,853-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:31:32,207-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:31:42,215-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:31:52,224-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:32:01,284-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:32:02,232-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:32:04,171-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:32:12,241-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:32:22,249-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:32:32,257-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:32:37,758-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 20:32:41,489-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 20:32:42,265-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:32:52,274-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:33:02,283-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:33:12,297-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:33:14,232-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:33:18,807-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:33:22,305-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:33:32,313-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:33:42,322-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 20:33:50,707-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 20:33:52,330-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:33:56,125-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:34:02,338-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:34:12,346-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:34:22,353-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:34:27,180-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:34:32,361-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:34:33,443-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:34:42,369-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:34:52,376-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:35:02,384-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 20:35:03,655-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 20:35:10,761-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 20:35:12,392-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:35:22,401-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:35:32,408-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 20:35:40,129-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 20:35:42,416-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 20:35:48,080-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 20:35:52,424-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:36:02,432-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:36:12,440-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:36:16,602-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:36:22,448-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:36:25,397-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:36:32,456-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:36:42,464-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:36:52,472-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:36:53,076-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:37:02,480-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:37:02,715-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:37:12,488-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:37:22,495-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 20:37:29,551-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 20:37:32,503-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:37:40,033-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:37:42,511-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:37:52,519-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:38:02,527-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 20:38:06,024-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 20:38:12,535-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:38:17,351-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:38:22,542-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:38:32,550-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 20:38:42,497-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 20:38:42,558-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:38:52,566-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:38:54,669-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:39:02,574-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:39:12,582-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 20:39:18,972-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 20:39:22,589-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:39:31,987-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:39:32,597-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:39:42,605-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:39:52,612-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:39:55,446-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:40:02,619-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:40:09,305-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:40:12,628-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:40:22,636-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 20:40:31,920-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 20:40:32,644-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:40:42,653-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 20:40:46,623-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 20:40:52,661-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:41:02,669-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 20:41:08,394-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 20:41:12,677-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:41:22,685-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:41:23,942-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:41:32,694-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:41:42,702-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:41:44,868-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:41:52,710-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:42:01,260-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:42:02,718-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:42:12,727-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 20:42:21,342-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 20:42:22,735-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:42:32,743-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 20:42:38,578-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 20:42:42,752-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:42:52,760-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 20:42:57,815-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 20:43:02,768-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:43:12,777-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:43:15,896-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:43:22,785-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:43:32,793-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 20:43:34,290-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 20:43:42,802-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:43:52,810-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 20:43:53,214-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 20:44:02,818-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-2013-11-07 20:44:10,766-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-No streams
-2013-11-07 20:44:12,826-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:44:22,834-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:44:30,533-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:44:32,842-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:44:42,850-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 20:44:47,238-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 20:44:52,858-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:45:02,867-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:45:07,851-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:45:12,875-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:45:22,883-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 20:45:23,712-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 20:45:32,891-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:45:42,899-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:45:45,169-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:45:52,907-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:46:00,187-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:46:02,915-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:46:12,923-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 20:46:22,487-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 20:46:22,931-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:46:32,939-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 20:46:36,661-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 20:46:42,947-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:46:52,955-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:46:59,805-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:47:02,963-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:47:12,971-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 20:47:13,135-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 20:47:22,979-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:47:32,987-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 20:47:37,123-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 20:47:42,995-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 20:47:49,610-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 20:47:53,003-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:48:03,011-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:48:13,019-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:48:14,441-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:48:23,027-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 20:48:26,084-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 20:48:33,035-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:48:43,043-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:48:51,759-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:48:53,051-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 20:49:02,559-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 20:49:03,059-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:49:13,067-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:49:23,075-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 20:49:29,078-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 20:49:33,083-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 20:49:39,033-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 20:49:43,091-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:49:53,099-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:50:03,107-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 20:50:06,395-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 20:50:13,114-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 20:50:15,508-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 20:50:23,122-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:50:33,130-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:50:43,138-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:50:43,713-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 20:50:51,981-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 20:50:53,145-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:51:03,153-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:51:13,161-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 20:51:21,031-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 20:51:23,169-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 20:51:28,455-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 20:51:33,177-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:51:43,184-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:51:53,192-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:51:58,349-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:52:03,200-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 20:52:04,929-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 20:52:13,208-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:52:23,215-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:52:33,223-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 20:52:35,668-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 20:52:41,403-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 20:52:43,231-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:52:53,239-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:53:03,247-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 20:53:12,986-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 20:53:13,255-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:53:17,877-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:53:23,263-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:53:33,271-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:53:43,279-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:53:50,304-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:53:53,287-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 20:53:54,352-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 20:54:03,295-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:54:13,303-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:54:23,311-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 20:54:27,622-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 20:54:30,825-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 20:54:33,318-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:54:43,326-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:54:53,333-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:55:03,341-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 20:55:04,940-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:55:07,300-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:55:13,349-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:55:23,357-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:55:33,364-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:55:42,258-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:55:43,372-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 20:55:43,774-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 20:55:53,380-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:56:03,388-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:56:13,396-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 20:56:19,576-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:56:20,248-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:56:23,403-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:56:33,411-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:56:43,419-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:56:53,427-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 20:56:56,722-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 20:56:56,894-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 20:57:03,435-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:57:13,443-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:57:23,451-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 20:57:33,195-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 20:57:33,458-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:57:34,212-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:57:43,466-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:57:53,474-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:58:03,482-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 20:58:09,670-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 20:58:11,530-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 20:58:13,490-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:58:23,498-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:58:33,506-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:58:43,514-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 20:58:46,144-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 20:58:48,848-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 20:58:53,522-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:59:03,530-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:59:13,538-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 20:59:22,618-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 20:59:23,546-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 20:59:26,166-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 20:59:33,554-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:59:43,562-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 20:59:53,570-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 20:59:59,091-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:00:03,484-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:00:03,578-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:00:13,586-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:00:23,594-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:00:33,602-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 21:00:35,567-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 21:00:40,802-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 21:00:43,610-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:00:53,618-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:01:03,626-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:01:12,040-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:01:13,634-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 21:01:18,120-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 21:01:23,642-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:01:33,650-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:01:43,658-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:01:48,514-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:01:53,666-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:01:55,438-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:02:03,673-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:02:13,681-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:02:23,689-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:02:24,988-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 21:02:32,756-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 21:02:33,697-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:02:43,705-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:02:53,713-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:03:01,462-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:03:03,721-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:03:10,074-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:03:13,728-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:03:23,736-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:03:33,744-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:03:37,936-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:03:43,752-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:03:47,392-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:03:53,760-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:04:03,768-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:04:13,776-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:04:14,410-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:04:23,783-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 21:04:24,710-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 21:04:33,791-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:04:43,799-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:04:50,884-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:04:53,807-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 21:05:02,029-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 21:05:03,815-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:05:13,823-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:05:23,831-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:05:27,358-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:05:33,839-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:05:39,346-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:05:43,847-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:05:53,855-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:06:03,832-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:06:03,863-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:06:13,871-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:06:16,664-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:06:23,879-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:06:33,887-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:06:40,306-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:06:43,895-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:06:53,903-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:06:53,982-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:07:03,911-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:07:13,918-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:07:16,780-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:07:23,927-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:07:31,300-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:07:33,935-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:07:43,943-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:07:53,254-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:07:53,951-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:08:03,959-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 21:08:08,618-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 21:08:13,967-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:08:23,975-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:08:29,728-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:08:33,983-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:08:43,991-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:08:45,936-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:08:53,999-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:09:04,007-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 21:09:06,203-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 21:09:14,016-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:09:23,254-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:09:24,024-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:09:34,032-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 21:09:42,677-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 21:09:44,040-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:09:54,048-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 21:10:00,572-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 21:10:04,056-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:10:14,064-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:10:19,150-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:10:24,071-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:10:34,079-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 21:10:37,891-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 21:10:44,088-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:10:54,095-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 21:10:55,625-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 21:11:04,102-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:11:14,110-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:11:15,208-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:11:24,117-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:11:32,099-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:11:34,125-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:11:44,133-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 21:11:52,527-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 21:11:54,141-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:12:04,149-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 21:12:08,573-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 21:12:14,157-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:12:24,165-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:12:29,845-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:12:34,173-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:12:44,181-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:12:45,047-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:12:54,189-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:13:04,197-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:13:07,163-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:13:14,205-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:13:21,521-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:13:24,213-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:13:34,221-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:13:44,230-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:13:44,481-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:13:54,237-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:13:57,995-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:14:04,246-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:14:14,254-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:14:21,799-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:14:24,263-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:14:34,271-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:14:34,469-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:14:44,279-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:14:54,288-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:14:59,117-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:15:04,296-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:15:10,943-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:15:14,304-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:15:24,312-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:15:34,321-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:15:36,435-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:15:44,329-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:15:47,417-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:15:54,338-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:16:04,346-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:16:13,753-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:16:14,354-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 21:16:23,890-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 21:16:24,362-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:16:34,370-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:16:44,379-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:16:51,071-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:16:54,387-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:17:00,365-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:17:04,395-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:17:14,403-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:17:24,412-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 21:17:28,391-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 21:17:34,420-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:17:36,839-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:17:44,428-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:17:54,436-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:18:04,445-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:18:05,708-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:18:13,313-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:18:14,453-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:18:24,461-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:18:34,469-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:18:43,026-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:18:44,478-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 21:18:49,786-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 21:18:54,486-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:19:04,493-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:19:14,502-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:19:20,344-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:19:24,509-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:19:26,261-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:19:34,518-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:19:44,526-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:19:54,534-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:19:57,662-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:20:02,736-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:20:04,542-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:20:14,550-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:20:24,558-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:20:34,567-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 21:20:34,981-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:20:39,209-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:20:44,575-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:20:54,583-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:21:04,592-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:21:12,298-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:21:14,600-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 21:21:15,684-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 21:21:24,608-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:21:34,617-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:21:44,625-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:21:49,616-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:21:52,158-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:21:54,633-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:22:04,641-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:22:14,649-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:22:24,657-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:22:26,934-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:22:28,633-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:22:34,665-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:22:44,674-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:22:54,682-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:23:04,252-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:23:04,690-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:23:05,107-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:23:14,698-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:23:24,707-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:23:34,715-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:23:41,570-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:23:41,581-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:23:44,723-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:23:54,732-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:24:04,740-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:24:14,748-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:24:18,055-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:24:18,888-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:24:24,757-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:24:34,765-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:24:44,773-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:24:54,529-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:24:54,781-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:24:56,206-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:25:04,790-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:25:14,798-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:25:24,806-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:25:31,003-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:25:33,524-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:25:34,814-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:25:44,822-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:25:54,830-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:26:04,838-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:26:07,477-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:26:10,842-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:26:14,846-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:26:24,853-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:26:34,861-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:26:43,951-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:26:44,869-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:26:48,160-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:26:54,877-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:27:04,885-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:27:14,893-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:27:20,425-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:27:24,901-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:27:25,478-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:27:34,909-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:27:44,917-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:27:54,925-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:27:56,899-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 21:28:02,796-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 21:28:04,933-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:28:14,941-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:28:24,949-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:28:33,373-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:28:34,956-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:28:40,114-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:28:44,964-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:28:54,972-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:29:04,980-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 21:29:09,847-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 21:29:14,988-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:29:17,432-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:29:24,996-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:29:35,004-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:29:45,007-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:29:46,321-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 21:29:54,751-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 21:29:55,015-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:30:05,023-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:30:15,030-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 21:30:22,796-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 21:30:25,039-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 21:30:32,070-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 21:30:35,046-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:30:45,054-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:30:55,063-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 4ms
-2013-11-07 21:30:59,272-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 4ms
-No streams
-2013-11-07 21:31:05,071-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:31:09,387-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:31:15,079-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:31:25,086-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:31:35,095-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:31:35,743-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:31:45,103-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 21:31:46,706-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 21:31:55,111-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:32:05,119-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:32:12,217-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:32:15,127-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:32:24,023-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:32:25,135-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:32:35,143-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:32:45,151-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:32:48,691-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:32:55,159-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:33:01,341-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:33:05,168-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:33:15,176-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 21:33:25,165-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 21:33:25,184-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:33:35,192-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:33:38,659-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:33:45,200-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:33:55,209-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:34:01,639-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:34:05,217-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:34:15,225-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:34:15,977-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:34:25,233-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:34:35,241-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:34:38,113-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:34:45,249-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:34:53,295-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:34:55,256-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:35:05,263-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 21:35:14,588-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 21:35:15,271-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:35:25,279-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 21:35:30,614-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 21:35:35,287-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:35:45,295-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:35:51,061-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:35:55,302-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:36:05,310-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:36:07,931-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:36:15,318-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:36:25,325-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 21:36:27,536-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 21:36:35,333-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:36:45,249-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:36:45,341-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:36:55,348-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:37:04,009-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:37:05,356-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:37:15,364-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:37:22,567-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:37:25,372-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:37:35,379-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:37:40,483-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:37:45,387-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:37:55,395-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:37:59,885-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:38:05,403-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:38:15,410-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:38:16,957-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:38:25,418-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:38:35,426-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 21:38:37,204-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 21:38:45,434-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 21:38:53,432-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 21:38:55,441-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:39:05,449-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:39:14,521-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:39:15,457-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:39:25,465-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:39:29,905-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:39:35,473-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:39:45,481-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 21:39:51,840-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 21:39:55,488-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:40:05,496-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:40:06,379-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:40:15,504-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:40:25,512-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 21:40:29,159-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 21:40:35,519-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:40:42,853-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:40:45,527-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:40:55,535-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:41:05,542-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:41:06,476-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:41:15,550-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 21:41:19,328-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 21:41:25,558-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:41:35,565-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:41:43,794-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:41:45,573-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:41:55,581-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:41:55,801-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:42:05,589-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:42:15,598-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:42:21,112-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:42:25,606-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 21:42:32,276-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 21:42:35,613-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:42:45,622-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:42:55,630-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 21:42:58,431-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 21:43:05,638-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:43:08,749-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:43:15,646-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:43:25,654-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:43:35,662-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:43:35,749-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:43:45,223-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:43:45,670-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:43:55,678-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:44:05,686-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:44:13,067-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:44:15,695-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:44:21,697-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:44:25,703-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:44:35,711-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:44:45,720-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:44:50,385-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:44:55,728-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 21:44:58,172-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 21:45:05,736-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:45:15,745-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:45:25,753-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:45:27,703-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:45:34,645-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:45:35,761-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:45:45,769-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:45:55,778-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 21:46:05,021-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 21:46:05,786-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:46:11,119-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:46:15,794-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:46:25,802-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:46:35,811-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:46:42,339-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:46:45,819-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:46:47,593-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:46:55,827-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:47:05,836-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:47:15,844-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:47:19,657-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 21:47:24,068-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 21:47:25,852-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:47:35,861-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:47:45,869-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:47:55,877-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:47:56,975-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:48:00,541-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:48:05,885-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:48:15,894-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:48:25,902-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:48:34,293-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:48:35,911-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 21:48:37,016-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 21:48:45,919-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:48:55,927-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:49:05,935-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 21:49:11,611-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:49:13,489-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:49:15,943-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:49:25,952-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:49:35,960-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:49:45,968-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:49:48,929-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 21:49:49,964-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 21:49:55,976-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:50:05,985-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:50:15,993-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:50:26,001-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:50:26,247-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:50:26,437-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:50:36,009-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:50:46,017-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:50:56,026-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 21:51:02,912-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 21:51:03,565-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 21:51:06,034-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:51:16,042-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:51:26,051-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:51:36,059-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:51:39,385-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:51:40,884-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:51:46,067-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:51:56,076-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:52:06,084-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 21:52:15,860-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 21:52:16,092-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:52:18,202-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:52:26,100-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:52:36,109-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:52:46,117-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 21:52:52,334-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:52:55,520-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:52:56,126-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:53:06,134-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:53:16,142-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:53:26,151-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:53:28,807-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:53:32,838-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:53:36,159-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:53:46,167-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:53:56,175-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:54:05,281-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:54:06,184-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:54:10,156-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:54:16,192-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:54:26,201-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:54:36,209-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 21:54:41,756-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 21:54:46,217-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 21:54:47,475-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 21:54:56,225-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:55:06,234-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:55:16,242-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:55:18,229-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:55:24,792-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:55:26,250-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:55:36,258-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:55:46,266-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:55:54,703-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:55:56,275-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:56:02,110-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:56:06,283-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:56:16,291-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:56:26,299-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:56:31,177-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:56:36,307-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:56:39,428-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:56:46,315-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:56:56,323-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:57:06,331-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:57:07,651-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:57:16,339-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:57:16,746-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:57:26,347-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:57:36,355-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:57:44,125-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:57:46,363-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:57:54,064-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:57:56,371-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:58:06,379-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:58:16,387-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 21:58:20,599-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 21:58:26,395-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:58:31,382-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:58:36,403-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:58:46,411-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:58:56,419-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:58:57,073-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:59:06,427-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 21:59:08,699-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 21:59:16,435-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:59:26,443-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:59:33,547-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:59:36,451-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 21:59:46,018-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 21:59:46,459-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 21:59:56,467-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:00:06,475-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:00:10,021-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:00:16,483-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:00:23,336-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:00:26,491-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:00:36,498-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:00:46,495-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:00:46,506-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:00:56,514-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 22:01:00,655-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 22:01:06,522-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:01:16,530-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 22:01:22,970-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 22:01:26,538-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:01:36,546-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:01:37,973-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:01:46,554-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:01:56,562-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:01:59,443-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:02:06,570-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:02:15,291-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:02:16,578-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:02:26,586-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:02:35,917-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:02:36,594-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:02:46,602-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 22:02:52,610-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 22:02:56,610-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:03:06,618-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:03:12,391-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:03:16,626-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:03:26,634-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:03:29,927-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:03:36,642-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:03:46,650-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:03:48,865-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:03:56,658-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:04:06,666-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 22:04:07,245-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 22:04:16,674-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:04:25,339-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:04:26,682-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:04:36,690-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 22:04:44,564-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 22:04:46,698-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:04:56,706-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 22:05:01,814-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 22:05:06,714-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:05:16,722-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:05:21,881-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:05:26,731-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:05:36,739-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 22:05:38,288-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 22:05:46,747-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:05:56,755-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:05:59,199-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:06:06,763-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:06:14,761-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:06:16,771-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:06:26,779-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:06:36,517-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:06:36,787-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:06:46,794-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:06:51,235-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:06:56,802-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:07:06,810-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 22:07:13,835-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 22:07:16,818-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:07:26,826-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 22:07:27,709-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 22:07:36,834-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:07:46,842-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 22:07:51,154-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 22:07:56,849-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:08:04,183-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:08:06,857-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:08:16,865-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:08:26,873-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:08:28,472-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:08:36,880-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 22:08:40,656-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 22:08:46,888-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:08:56,896-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:09:05,790-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:09:06,903-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:09:16,911-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:09:17,131-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:09:26,919-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:09:36,927-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:09:43,108-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:09:46,935-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:09:53,605-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:09:56,943-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:10:06,951-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:10:16,959-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:10:20,426-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:10:26,967-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:10:30,079-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:10:36,975-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:10:46,983-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:10:56,991-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 22:10:57,744-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:11:06,553-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:11:06,999-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:11:17,007-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:11:27,015-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:11:35,062-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:11:37,023-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 22:11:43,028-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 22:11:47,031-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:11:57,039-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:12:07,047-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:12:12,380-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:12:17,055-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:12:19,501-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:12:27,063-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:12:37,071-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:12:47,079-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 22:12:49,699-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 22:12:55,976-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 22:12:57,087-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:13:07,095-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:13:17,103-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:13:27,016-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:13:27,111-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 22:13:32,449-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 22:13:37,119-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:13:47,127-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:13:57,135-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:14:04,334-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:14:07,144-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 22:14:08,924-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 22:14:17,152-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:14:27,160-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:14:37,168-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 22:14:41,653-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:14:45,397-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:14:47,176-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:14:57,184-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:15:07,192-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:15:17,200-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:15:18,970-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:15:21,871-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:15:27,208-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:15:37,216-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:15:47,224-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:15:56,288-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:15:57,232-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:15:58,345-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:16:07,240-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:16:17,249-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:16:27,257-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:16:33,606-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 22:16:34,820-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 22:16:37,265-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:16:47,273-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:16:57,281-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:17:07,289-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:17:10,924-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:17:11,294-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:17:17,297-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:17:27,306-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:17:37,314-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:17:47,322-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:17:47,768-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 22:17:48,242-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 22:17:57,330-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:18:07,338-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:18:17,346-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:18:24,242-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:18:25,560-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:18:27,353-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:18:37,361-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:18:47,369-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:18:57,377-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:19:00,717-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:19:02,878-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:19:07,385-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:19:17,393-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:19:27,400-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 4ms
-2013-11-07 22:19:37,193-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 4ms
-No streams
-2013-11-07 22:19:37,408-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:19:40,196-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:19:47,416-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:19:57,424-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:20:07,432-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 22:20:13,665-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 22:20:17,440-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:20:17,514-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:20:27,448-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:20:37,456-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:20:47,464-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 22:20:50,139-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 22:20:54,833-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 22:20:57,472-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:21:07,479-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:21:17,487-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 22:21:26,613-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 22:21:27,495-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:21:32,151-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:21:37,503-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:21:47,511-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:21:57,519-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 22:22:03,086-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 22:22:07,526-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:22:09,469-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:22:17,534-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:22:27,542-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:22:37,550-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 22:22:39,561-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:22:46,787-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:22:47,559-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:22:57,566-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:23:07,574-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:23:16,034-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:23:17,583-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:23:24,105-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:23:27,591-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:23:37,599-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:23:47,607-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 22:23:52,509-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 22:23:57,615-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 22:24:01,423-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 22:24:07,623-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:24:17,631-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:24:27,639-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 22:24:28,983-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 22:24:37,647-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:24:38,741-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:24:47,655-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:24:57,663-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 22:25:05,457-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 22:25:07,672-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:25:16,059-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:25:17,680-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:25:27,688-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:25:37,696-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 22:25:41,931-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 22:25:47,704-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 22:25:53,378-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 22:25:57,712-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:26:07,720-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:26:17,728-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 22:26:18,405-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 22:26:27,736-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:26:30,695-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:26:37,744-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:26:47,752-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:26:54,878-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:26:57,761-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:27:07,769-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 22:27:08,015-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 22:27:17,777-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:27:27,785-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:27:31,352-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:27:37,793-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:27:45,332-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:27:47,802-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:27:57,810-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:28:07,818-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 22:28:07,826-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 22:28:17,826-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 5ms, rtt: 5ms
-2013-11-07 22:28:22,654-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 5ms, rtt: 5ms
-No streams
-2013-11-07 22:28:27,834-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:28:37,842-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 22:28:44,301-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 22:28:47,851-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:28:57,859-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:28:59,968-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:29:07,867-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:29:17,875-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 22:29:20,774-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 22:29:27,883-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:29:37,286-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:29:37,891-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:29:47,899-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:29:57,248-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:29:57,907-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:30:07,915-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:30:14,604-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:30:17,923-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:30:27,931-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 22:30:33,722-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 22:30:37,939-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:30:47,948-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:30:51,922-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:30:57,956-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:31:07,964-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 22:31:10,196-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 22:31:17,972-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:31:27,980-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 22:31:29,241-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 22:31:37,988-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 22:31:46,671-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 22:31:47,996-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:31:58,003-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:32:06,558-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:32:08,012-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:32:18,019-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:32:23,145-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:32:28,027-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:32:38,035-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:32:43,876-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:32:48,043-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:32:58,051-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 22:32:59,620-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 22:33:08,060-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:33:18,068-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:33:21,194-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:33:28,076-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 22:33:36,093-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 22:33:38,084-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:33:48,092-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:33:58,101-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 22:33:58,512-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 22:34:08,109-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 22:34:12,568-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 22:34:18,117-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:34:28,125-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:34:35,831-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:34:38,133-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:34:48,141-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 22:34:49,041-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 22:34:58,149-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:35:08,157-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:35:13,149-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:35:18,165-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:35:25,515-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:35:28,173-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:35:38,181-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:35:48,189-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 22:35:50,468-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 22:35:58,197-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 22:36:01,989-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 22:36:08,205-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:36:18,213-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:36:27,785-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:36:28,221-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:36:38,229-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:36:38,463-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:36:48,237-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:36:58,245-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 22:37:05,104-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 22:37:08,253-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 22:37:14,937-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 22:37:18,261-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:37:28,269-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:37:38,278-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:37:42,421-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:37:48,286-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 22:37:51,411-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 22:37:58,294-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:38:08,302-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:38:18,310-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 22:38:19,740-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 22:38:27,885-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 22:38:28,320-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:38:38,327-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:38:48,336-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:38:57,057-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:38:58,343-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:39:04,359-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:39:08,351-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:39:18,359-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:39:28,367-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 22:39:34,375-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 22:39:38,375-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 22:39:40,834-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 22:39:48,383-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:39:58,391-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:40:08,399-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:40:11,693-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:40:17,307-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:40:18,407-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:40:28,416-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:40:38,424-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:40:48,432-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 22:40:49,011-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:40:53,781-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:40:58,440-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:41:08,448-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:41:18,456-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 22:41:26,330-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 22:41:28,464-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 22:41:30,255-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 22:41:38,472-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:41:48,480-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:41:58,488-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 22:42:03,647-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 22:42:06,730-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 22:42:08,496-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:42:18,504-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:42:28,512-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:42:38,520-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:42:40,965-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 22:42:43,203-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 22:42:48,528-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:42:58,535-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:43:08,544-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:43:18,283-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:43:18,552-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 22:43:19,677-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 22:43:28,560-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:43:38,568-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:43:48,576-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:43:55,601-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 22:43:56,151-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 22:43:58,584-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:44:08,592-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:44:18,600-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:44:28,608-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 22:44:32,625-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:44:32,919-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:44:38,615-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:44:48,623-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:44:58,631-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:45:08,639-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:45:09,099-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 22:45:10,237-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 22:45:18,647-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:45:28,655-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:45:38,663-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 22:45:45,574-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 22:45:47,556-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 22:45:48,671-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:45:58,679-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:46:08,687-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:46:18,695-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 22:46:22,048-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:46:24,873-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:46:28,703-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:46:38,711-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:46:48,719-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 22:46:58,522-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 22:46:58,727-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 22:47:02,191-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 22:47:08,736-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:47:18,744-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:47:28,752-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 22:47:34,996-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 22:47:38,760-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:47:39,509-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:47:48,768-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:47:58,776-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:48:08,784-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 22:48:11,471-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:48:16,827-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:48:18,792-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:48:28,800-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:48:38,807-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 22:48:47,944-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 22:48:48,815-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 22:48:54,146-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 22:48:58,823-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:49:08,831-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:49:18,839-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 22:49:24,418-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 22:49:28,847-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:49:31,463-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:49:38,855-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:49:48,863-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:49:58,871-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 22:50:00,892-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 22:50:08,780-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 22:50:08,879-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:50:18,887-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:50:28,895-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:50:37,366-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:50:38,903-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:50:46,099-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:50:48,911-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:50:58,919-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:51:08,927-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 22:51:13,840-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 22:51:18,935-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:51:23,417-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:51:28,943-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:51:38,952-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:51:48,960-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:51:50,314-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:51:58,968-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 22:52:00,736-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 22:52:08,976-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:52:18,984-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 22:52:26,789-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 22:52:28,993-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:52:38,053-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:52:39,001-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:52:49,009-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:52:59,017-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:53:03,263-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:53:09,025-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 22:53:15,371-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 22:53:19,033-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:53:29,042-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:53:39,050-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 22:53:39,738-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 22:53:49,058-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:53:52,689-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:53:59,066-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:54:09,074-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:54:16,211-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:54:19,082-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:54:29,091-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:54:30,007-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:54:39,099-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:54:49,107-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 22:54:52,686-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 22:54:59,115-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 22:55:07,326-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 22:55:09,124-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:55:19,132-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:55:29,140-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:55:29,159-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:55:39,149-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:55:44,643-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:55:49,157-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:55:59,165-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 22:56:05,634-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 22:56:09,173-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:56:19,182-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 22:56:21,962-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 22:56:29,190-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:56:39,198-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:56:42,108-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:56:49,206-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:56:59,214-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:56:59,280-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:57:09,223-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 22:57:18,583-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 22:57:19,231-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:57:29,239-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 22:57:36,598-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 22:57:39,248-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:57:49,256-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 22:57:55,056-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 22:57:59,264-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:58:09,272-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:58:13,916-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:58:19,280-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:58:29,289-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 22:58:31,531-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 22:58:39,297-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:58:49,305-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 22:58:51,234-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 22:58:59,314-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:59:08,004-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:59:09,322-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:59:19,331-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:59:28,552-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:59:29,339-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:59:39,347-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 22:59:44,478-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 22:59:49,355-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 22:59:59,363-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 23:00:05,871-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 23:00:09,373-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:00:19,381-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:00:20,952-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:00:29,389-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:00:39,397-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:00:43,189-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:00:49,406-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 23:00:57,427-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 23:00:59,414-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:01:09,422-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:01:19,430-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:01:20,507-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:01:29,438-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:01:33,900-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:01:39,447-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:01:49,462-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:01:57,825-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:01:59,470-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:02:09,478-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 23:02:10,375-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 23:02:19,486-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:02:29,494-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:02:35,143-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:02:39,502-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:02:46,848-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:02:49,510-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:02:59,519-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:03:09,527-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:03:12,461-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:03:19,535-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:03:23,322-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:03:29,543-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:03:39,552-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:03:49,560-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:03:49,779-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:03:59,568-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 23:03:59,797-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 23:04:09,576-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:04:19,586-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:04:27,097-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:04:29,594-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:04:36,270-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:04:39,602-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:04:49,610-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:04:59,618-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 23:05:04,416-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 23:05:09,626-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 23:05:12,745-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 23:05:19,635-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:05:29,643-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:05:39,651-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:05:41,733-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:05:49,218-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:05:49,660-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:05:59,668-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:06:09,677-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:06:19,051-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:06:19,685-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:06:25,693-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:06:29,693-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:06:39,701-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:06:49,710-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:06:56,369-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:06:59,718-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:07:02,167-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:07:09,726-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:07:19,735-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:07:29,743-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:07:33,687-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 23:07:38,641-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 23:07:39,751-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:07:49,759-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:07:59,767-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:08:09,775-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 23:08:11,006-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:08:15,116-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:08:19,783-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:08:29,792-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:08:39,800-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:08:48,323-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:08:49,808-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 23:08:51,591-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 23:08:59,817-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:09:09,825-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:09:19,833-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:09:25,642-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:09:28,064-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:09:29,842-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:09:39,850-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:09:49,858-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:09:59,866-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:10:02,960-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 23:10:04,539-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 23:10:09,874-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:10:19,883-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:10:29,891-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:10:39,900-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:10:40,278-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:10:41,012-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:10:49,908-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:10:59,917-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:11:09,926-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 23:11:17,487-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:11:17,597-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:11:19,934-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:11:29,942-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:11:39,950-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:11:49,958-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:11:53,960-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:11:54,915-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:11:59,967-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:12:09,975-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:12:19,983-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:12:29,991-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:12:30,434-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:12:32,233-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:12:39,999-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:12:50,008-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:13:00,016-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:13:06,908-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:13:09,551-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:13:10,024-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:13:20,032-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:13:30,041-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:13:40,049-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:13:43,382-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:13:46,869-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:13:50,057-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:14:00,066-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:14:10,074-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 23:14:19,855-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 23:14:20,083-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:14:24,187-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:14:30,092-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:14:40,100-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:14:50,108-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 23:14:56,331-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 23:15:00,116-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:15:01,505-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:15:10,123-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:15:20,131-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:15:30,139-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:15:32,804-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:15:38,823-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:15:40,147-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:15:50,154-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:16:00,162-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 23:16:09,279-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 23:16:10,170-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:16:16,141-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:16:20,178-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:16:30,186-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:16:40,194-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 23:16:45,752-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-07 23:16:50,201-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:16:53,459-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:17:00,209-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:17:10,217-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:17:20,224-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:17:22,226-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:17:30,232-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 23:17:30,778-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 23:17:40,240-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:17:50,248-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:17:58,700-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:18:00,256-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:18:08,095-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:18:10,263-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:18:20,271-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:18:30,279-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:18:35,174-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:18:40,287-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 23:18:45,414-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 23:18:50,294-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:19:00,302-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:19:10,310-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 23:19:11,650-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 23:19:20,318-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:19:22,731-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:19:30,326-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:19:40,334-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:19:48,123-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:19:50,341-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:20:00,049-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:20:00,349-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:20:10,357-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:20:20,365-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:20:24,598-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:20:30,373-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:20:37,367-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:20:40,381-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:20:50,389-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:21:00,398-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:21:01,072-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:21:10,405-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:21:14,685-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:21:20,413-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:21:30,421-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:21:37,547-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:21:40,429-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:21:50,437-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:21:52,003-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:22:00,445-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:22:10,452-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:22:14,021-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:22:20,460-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 23:22:29,322-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 23:22:30,468-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:22:40,476-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:22:50,484-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:22:50,495-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:23:00,492-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 23:23:06,640-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 23:23:10,500-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:23:20,507-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:23:26,969-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:23:30,516-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:23:40,523-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:23:43,957-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:23:50,531-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:24:00,539-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 23:24:03,444-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 23:24:10,547-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:24:20,556-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:24:21,275-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:24:30,563-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:24:39,917-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:24:40,571-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:24:50,579-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:24:58,593-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:25:00,586-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:25:10,594-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:25:16,391-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:25:20,602-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:25:30,610-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:25:35,911-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:25:40,617-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:25:50,625-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 23:25:52,866-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 23:26:00,634-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:26:10,642-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 23:26:13,230-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 23:26:20,650-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 23:26:29,340-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 23:26:30,657-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:26:40,665-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:26:50,547-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:26:50,673-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:27:00,681-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:27:05,813-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:27:10,688-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:27:20,696-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:27:27,865-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:27:30,704-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:27:40,712-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 23:27:42,288-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 23:27:50,720-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:28:00,728-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:28:05,183-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:28:10,735-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 23:28:18,762-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 23:28:20,743-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:28:30,751-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:28:40,759-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:28:42,501-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:28:50,767-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 23:28:55,236-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 23:29:00,776-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:29:10,783-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:29:19,819-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:29:20,793-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:29:30,800-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:29:31,709-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:29:40,809-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:29:50,817-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:29:57,137-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:30:00,826-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 23:30:08,184-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 23:30:10,834-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:30:20,842-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:30:30,851-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:30:34,455-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:30:40,861-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 23:30:44,658-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 23:30:50,869-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:31:00,877-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:31:10,885-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-07 23:31:11,774-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-07 23:31:20,893-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:31:21,131-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:31:30,901-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:31:40,909-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:31:49,091-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:31:50,916-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 23:31:57,606-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 23:32:00,924-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:32:10,931-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:32:20,939-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:32:26,409-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:32:30,947-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:32:34,079-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:32:40,954-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:32:50,962-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:33:00,969-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:33:03,727-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:33:10,553-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:33:10,977-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:33:20,984-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:33:30,994-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:33:41,001-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:33:41,045-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:33:47,027-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:33:51,009-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:34:01,016-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:34:11,024-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 23:34:18,364-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 23:34:21,032-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:34:23,501-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:34:31,040-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:34:41,050-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:34:51,058-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 23:34:55,682-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:34:59,975-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:35:01,070-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:35:11,080-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:35:21,089-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:35:31,097-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 23:35:33,000-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:35:36,449-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:35:41,105-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:35:51,114-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:36:01,123-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 23:36:10,318-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 23:36:11,131-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:36:12,923-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:36:21,139-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:36:31,147-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:36:41,155-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:36:47,635-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:36:49,397-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:36:51,163-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:37:01,171-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:37:11,179-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:37:21,187-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 23:37:24,954-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:37:25,871-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:37:31,195-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:37:41,203-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:37:51,210-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:38:01,218-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:38:02,271-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:38:02,345-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:38:11,226-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:38:21,234-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:38:31,242-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:38:38,819-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:38:39,589-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:38:41,250-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:38:51,258-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:39:01,266-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:39:11,274-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 23:39:15,294-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:39:16,907-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:39:21,282-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:39:31,290-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:39:41,304-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:39:51,312-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 23:39:51,768-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:39:54,225-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:40:01,320-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:40:11,329-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:40:21,337-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 23:40:28,242-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 23:40:31,345-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:40:31,543-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:40:41,353-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:40:51,361-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:41:01,370-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-07 23:41:04,715-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:41:08,861-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:41:11,378-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:41:21,386-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:41:31,394-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:41:41,189-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:41:41,402-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:41:46,179-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:41:51,410-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:42:01,418-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:42:11,427-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 23:42:17,664-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 23:42:21,434-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:42:23,497-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:42:31,443-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:42:41,451-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:42:51,459-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:42:54,137-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:43:00,815-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:43:01,467-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:43:11,475-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:43:21,483-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:43:30,611-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:43:31,491-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 23:43:38,134-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 23:43:41,499-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:43:51,507-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:44:01,516-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:44:07,085-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:44:11,524-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:44:15,451-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:44:21,532-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:44:31,540-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:44:41,548-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:44:43,559-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:44:51,556-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:44:52,769-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:45:01,564-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:45:11,573-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 23:45:20,034-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 23:45:21,581-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 23:45:30,088-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 23:45:31,589-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:45:41,597-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:45:51,605-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 23:45:56,508-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 23:46:01,613-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:46:07,405-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:46:11,621-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:46:21,630-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:46:31,638-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:46:32,981-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:46:41,646-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:46:44,723-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:46:51,654-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:47:01,661-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 23:47:09,456-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 23:47:11,669-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:47:21,677-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:47:22,041-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:47:31,685-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:47:41,693-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:47:45,929-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:47:51,701-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:47:59,359-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:48:01,709-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:48:11,717-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:48:21,725-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:48:22,403-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:48:31,734-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:48:36,677-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:48:41,742-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:48:51,750-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:48:58,877-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:49:01,758-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:49:11,766-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:49:13,995-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:49:21,773-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:49:31,781-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:49:35,351-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:49:41,789-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:49:51,313-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:49:51,797-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:50:01,805-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:50:11,813-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:50:11,825-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:50:21,821-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:50:28,631-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:50:31,828-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:50:41,836-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:50:48,299-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:50:51,844-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:51:01,852-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 23:51:05,950-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 23:51:11,860-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:51:21,868-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:51:24,773-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:51:31,876-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:51:41,884-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:51:43,267-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:51:51,892-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:52:01,247-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:52:01,900-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:52:11,907-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:52:20,585-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:52:21,915-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:52:31,923-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:52:37,721-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:52:41,931-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:52:51,939-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:52:57,903-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:53:01,947-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:53:11,955-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 23:53:14,196-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 23:53:21,963-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:53:31,972-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:53:35,221-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:53:41,979-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:53:50,669-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:53:51,987-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:54:01,994-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:54:12,002-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 23:54:12,540-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 23:54:22,010-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:54:27,143-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:54:32,017-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:54:42,025-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:54:49,857-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:54:52,033-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:55:02,041-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:55:03,617-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:55:12,048-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:55:22,056-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:55:27,175-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:55:32,064-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:55:40,091-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:55:42,071-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:55:52,079-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:56:02,087-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:56:04,493-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:56:12,095-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:56:16,565-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:56:22,103-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:56:32,110-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:56:41,811-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:56:42,118-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:56:52,126-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:56:53,039-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:57:02,134-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:57:12,142-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:57:19,129-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:57:22,149-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:57:29,513-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:57:32,157-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:57:42,165-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:57:52,173-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:57:56,447-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:58:02,181-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 23:58:05,988-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 23:58:12,189-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:58:22,196-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:58:32,204-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:58:33,765-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:58:42,212-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:58:42,461-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-07 23:58:52,220-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:59:02,228-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-07 23:59:11,084-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-07 23:59:12,235-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:59:18,935-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:59:22,243-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:59:32,251-0600 INFO Multiplexer No streams
-No streams
-2013-11-07 23:59:42,259-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-07 23:59:48,401-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-07 23:59:52,267-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-07 23:59:55,409-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 00:00:02,275-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:00:12,283-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:00:22,291-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 00:00:25,720-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:00:31,883-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:00:32,298-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:00:42,306-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:00:52,314-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:01:02,322-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:01:03,037-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 00:01:08,357-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 00:01:12,330-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:01:22,338-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:01:32,346-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 00:01:40,356-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 00:01:42,355-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 00:01:44,831-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 00:01:52,367-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:02:02,379-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:02:12,390-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 00:02:17,674-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 00:02:21,306-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 00:02:22,400-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:02:32,409-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:02:42,420-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:02:52,431-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 00:02:54,992-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 00:02:57,779-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 00:03:02,439-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:03:12,450-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:03:22,458-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 00:03:32,310-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 00:03:32,471-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 00:03:34,254-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 00:03:42,482-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:03:52,495-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:04:02,505-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 00:04:09,629-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 00:04:10,728-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 00:04:12,509-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:04:22,523-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:04:32,533-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:04:42,543-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:04:46,946-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:04:47,201-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:04:52,554-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:05:02,564-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:05:12,574-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:05:22,583-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 00:05:23,675-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 00:05:24,265-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 00:05:32,593-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:05:42,603-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:05:52,614-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 00:06:00,150-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 00:06:01,583-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 00:06:02,624-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:06:12,634-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:06:22,644-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:06:32,653-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 00:06:36,624-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 00:06:38,901-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 00:06:42,664-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:06:52,674-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:07:02,684-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:07:12,694-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 00:07:13,097-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 00:07:16,219-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 00:07:22,702-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:07:32,710-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:07:42,718-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 00:07:49,572-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 00:07:52,727-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:07:53,537-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:08:02,735-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:08:12,743-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:08:22,751-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:08:26,045-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:08:30,855-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:08:32,759-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:08:42,767-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:08:52,776-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 00:09:02,520-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 00:09:02,784-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 00:09:08,174-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 00:09:12,792-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:09:22,800-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:09:32,808-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 00:09:38,993-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 00:09:42,817-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 00:09:45,492-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 00:09:52,825-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:10:02,833-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:10:12,841-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 00:10:15,468-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 00:10:22,810-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 00:10:22,850-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:10:32,858-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:10:42,866-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 00:10:51,942-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 00:10:52,874-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:11:00,127-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:11:02,882-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:11:12,890-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:11:22,898-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:11:28,415-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:11:32,906-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:11:37,446-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:11:42,914-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:11:52,922-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:12:02,931-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:12:04,889-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:12:12,939-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:12:14,764-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:12:22,947-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:12:32,955-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 00:12:41,364-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 00:12:42,965-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:12:52,082-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:12:52,973-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:13:02,981-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:13:12,989-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:13:17,837-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:13:22,998-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:13:29,400-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:13:33,006-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:13:43,014-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:13:53,022-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:13:54,311-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:14:03,030-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:14:06,718-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:14:13,039-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:14:23,047-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 00:14:30,785-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 00:14:33,055-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:14:43,064-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:14:44,036-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:14:53,072-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:15:03,080-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 00:15:07,260-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 00:15:13,088-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:15:21,354-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:15:23,096-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:15:33,105-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:15:43,113-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 00:15:43,733-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 00:15:53,121-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 00:15:58,672-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 00:16:03,131-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:16:13,139-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 00:16:20,208-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 00:16:23,147-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:16:33,156-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:16:35,990-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:16:43,164-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:16:53,172-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:16:56,681-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:17:03,181-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:17:13,190-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:17:13,308-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:17:23,198-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 00:17:33,156-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 00:17:33,206-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:17:43,215-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 00:17:50,627-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 00:17:53,223-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:18:03,231-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 00:18:09,629-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 00:18:13,240-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:18:23,248-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:18:27,944-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:18:33,256-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:18:43,264-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 00:18:46,104-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 00:18:53,273-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:19:03,281-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 00:19:05,262-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 00:19:13,290-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:19:22,577-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:19:23,298-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:19:33,306-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:19:42,580-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:19:43,315-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:19:53,323-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:19:59,051-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:20:03,332-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:20:13,340-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:20:19,898-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:20:23,348-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:20:33,356-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 00:20:35,526-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 00:20:43,364-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:20:53,372-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:20:57,216-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:21:03,380-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:21:12,000-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:21:13,388-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:21:23,396-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:21:33,404-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:21:34,534-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:21:43,412-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:21:48,474-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:21:53,420-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:22:03,429-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:22:11,852-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:22:13,437-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:22:23,444-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 00:22:24,948-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 00:22:33,452-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:22:43,460-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:22:49,170-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:22:53,468-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 00:23:01,423-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 00:23:03,476-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:23:13,484-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:23:23,492-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 00:23:26,489-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 00:23:33,500-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:23:37,896-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:23:43,508-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:23:53,516-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:24:03,524-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:24:03,806-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:24:13,532-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:24:14,370-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:24:23,540-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:24:33,548-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:24:41,124-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:24:43,556-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:24:50,844-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:24:53,564-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:25:03,572-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:25:13,580-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 00:25:18,443-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 00:25:23,588-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 00:25:27,319-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 00:25:33,596-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:25:43,604-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:25:53,612-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:25:55,760-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:26:03,620-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 00:26:03,792-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 00:26:13,628-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:26:23,636-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 00:26:33,079-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 00:26:33,645-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 00:26:40,267-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 00:26:43,653-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:26:53,661-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:27:03,669-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:27:10,396-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:27:13,677-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 00:27:16,740-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 00:27:23,685-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:27:33,693-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:27:43,701-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 00:27:47,715-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:27:53,214-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:27:53,709-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:28:03,718-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:28:13,726-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:28:23,734-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:28:25,032-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 00:28:29,688-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 00:28:33,742-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:28:43,750-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:28:53,758-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:29:02,350-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:29:03,766-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 4ms
-2013-11-08 00:29:06,165-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 4ms
-No streams
-2013-11-08 00:29:13,774-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:29:23,783-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:29:33,791-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:29:39,668-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:29:42,636-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:29:43,799-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:29:53,807-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:30:03,815-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:30:13,823-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:30:16,986-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:30:19,110-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:30:23,831-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:30:33,839-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:30:43,848-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:30:53,856-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:30:54,305-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:30:55,584-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:31:03,864-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:31:13,872-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:31:23,880-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:31:31,623-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 00:31:32,059-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 00:31:33,888-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:31:43,896-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:31:53,904-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:32:03,912-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:32:08,532-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:32:08,941-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:32:13,920-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:32:23,928-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:32:33,937-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:32:43,945-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:32:45,006-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:32:46,260-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:32:53,953-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:33:03,961-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:33:13,970-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 00:33:21,481-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:33:23,578-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:33:23,978-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:33:33,986-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:33:43,994-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:33:54,003-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:33:57,954-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:34:00,896-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:34:04,011-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:34:14,020-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:34:24,028-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:34:34,036-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 00:34:34,428-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 00:34:38,215-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 00:34:44,045-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:34:54,053-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:35:04,061-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:35:10,902-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:35:14,070-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:35:15,532-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:35:24,078-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:35:34,087-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:35:44,095-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:35:47,376-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:35:52,850-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:35:54,103-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:36:04,111-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:36:14,120-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:36:23,850-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:36:24,128-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:36:30,168-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:36:34,136-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:36:44,144-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:36:54,152-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:37:00,324-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:37:04,160-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:37:07,486-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:37:14,168-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:37:24,176-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:37:34,185-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 00:37:36,799-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 00:37:44,193-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 00:37:44,805-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 00:37:54,201-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:38:04,210-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:38:13,272-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:38:14,218-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:38:22,122-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:38:24,226-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:38:34,234-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:38:44,243-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:38:49,746-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:38:54,251-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:38:59,440-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:39:04,260-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:39:14,268-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:39:24,277-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:39:26,220-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:39:34,285-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:39:36,758-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:39:44,294-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:39:54,302-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 00:40:02,695-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 00:40:04,310-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:40:14,076-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:40:14,318-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:40:24,326-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:40:34,334-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:40:39,168-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:40:44,342-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 00:40:51,395-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 00:40:54,350-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:41:04,358-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:41:14,366-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:41:15,642-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:41:24,374-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:41:28,712-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:41:34,382-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:41:44,390-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:41:52,116-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:41:54,399-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:42:04,407-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:42:06,030-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:42:14,415-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:42:24,423-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 00:42:28,591-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 00:42:34,431-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 00:42:43,349-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 00:42:44,440-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:42:54,448-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:43:04,456-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:43:05,065-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:43:14,464-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 00:43:20,667-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 00:43:24,472-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:43:34,481-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 00:43:41,540-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 00:43:44,489-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:43:54,497-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:43:57,984-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:44:04,505-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:44:14,513-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:44:18,014-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:44:24,521-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:44:34,529-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:44:35,303-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:44:44,537-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 00:44:54,488-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 00:44:54,545-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:45:04,553-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:45:12,620-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:45:14,561-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:45:24,570-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:45:30,962-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:45:34,578-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:45:44,586-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 00:45:49,939-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 00:45:54,594-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:46:04,602-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:46:07,436-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:46:14,610-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:46:24,619-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:46:27,256-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:46:34,627-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:46:43,910-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:46:44,635-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:46:54,643-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:47:04,574-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:47:04,652-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:47:14,660-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 00:47:20,383-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 00:47:24,668-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:47:34,676-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:47:41,892-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:47:44,684-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:47:54,692-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:47:56,858-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:48:04,700-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:48:14,708-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:48:19,210-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:48:24,716-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 00:48:33,333-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 00:48:34,724-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:48:44,732-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:48:54,741-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:48:56,529-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:49:04,749-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:49:09,807-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:49:14,757-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:49:24,765-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:49:33,846-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:49:34,773-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:49:44,781-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 00:49:46,282-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 00:49:54,789-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:50:04,797-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 00:50:11,165-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 00:50:14,805-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:50:22,756-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:50:24,813-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:50:34,821-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:50:44,829-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:50:48,483-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:50:54,837-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:50:59,230-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:51:04,845-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:51:14,853-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:51:24,861-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:51:25,801-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:51:34,869-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:51:35,704-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:51:44,877-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:51:54,885-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 00:52:03,120-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 00:52:04,893-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:52:12,178-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:52:14,901-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:52:24,909-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:52:34,917-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:52:40,437-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:52:44,925-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:52:48,652-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:52:54,933-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:53:04,942-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:53:14,949-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 00:53:17,756-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 00:53:24,957-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 00:53:25,127-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 00:53:34,966-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:53:44,974-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:53:54,981-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:53:55,073-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:54:01,600-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:54:04,990-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:54:14,998-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:54:25,006-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:54:32,391-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:54:35,014-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 00:54:38,075-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 00:54:45,022-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:54:55,030-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:55:05,038-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 00:55:09,710-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 00:55:14,549-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 00:55:15,046-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:55:25,054-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:55:35,062-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:55:45,070-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 00:55:47,028-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 00:55:51,022-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 00:55:55,078-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:56:05,087-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:56:15,095-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 00:56:24,346-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 00:56:25,103-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 00:56:27,497-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 00:56:35,111-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:56:45,119-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:56:55,128-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:57:01,663-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 00:57:03,971-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 00:57:05,137-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:57:15,145-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:57:25,153-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:57:35,162-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 00:57:38,982-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:57:40,444-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:57:45,170-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:57:55,179-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:58:05,187-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:58:15,196-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:58:16,300-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:58:16,918-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:58:25,205-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:58:35,213-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:58:45,226-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:58:53,392-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 00:58:53,617-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 00:58:55,236-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:59:05,247-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:59:15,258-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:59:25,268-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 00:59:29,867-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 4ms
-2013-11-08 00:59:30,938-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 4ms
-No streams
-2013-11-08 00:59:35,280-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:59:45,292-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 00:59:55,303-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:00:05,314-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 01:00:06,341-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 01:00:08,254-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 01:00:15,326-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:00:25,337-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:00:35,351-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 01:00:42,815-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 01:00:45,371-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 01:00:45,572-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 01:00:55,391-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:01:05,411-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:01:15,412-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:01:19,289-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:01:22,890-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:01:25,417-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:01:35,422-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:01:45,423-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:01:55,434-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:01:55,762-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:02:00,208-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:02:05,444-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:02:15,456-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:02:25,465-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:02:32,237-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:02:35,474-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:02:37,527-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:02:45,484-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:02:55,494-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:03:05,503-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 01:03:08,711-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:03:14,845-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:03:15,513-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:03:25,523-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:03:35,532-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 01:03:45,185-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 01:03:45,542-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:03:52,163-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:03:55,552-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:04:05,563-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:04:15,574-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:04:21,658-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:04:25,584-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:04:29,481-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:04:35,592-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:04:45,601-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:04:55,610-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:04:58,133-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:05:05,618-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:05:06,798-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:05:15,630-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:05:25,641-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 01:05:34,607-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 01:05:35,653-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:05:44,116-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:05:45,665-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:05:55,676-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:06:05,688-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 01:06:11,081-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 01:06:15,697-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:06:21,435-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:06:25,707-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:06:35,719-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:06:45,731-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:06:47,554-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:06:55,743-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:06:58,753-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:07:05,756-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:07:15,768-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:07:24,029-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:07:25,781-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:07:35,794-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:07:36,071-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:07:45,806-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:07:55,819-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:08:00,504-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:08:05,832-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:08:13,390-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:08:15,842-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:08:25,852-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:08:35,861-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:08:36,977-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:08:45,874-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:08:50,707-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:08:55,886-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:09:05,896-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 01:09:13,452-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 01:09:15,906-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:09:25,915-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:09:28,025-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:09:35,924-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:09:45,936-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:09:49,926-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:09:55,947-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 01:10:05,344-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 01:10:05,960-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:10:15,972-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:10:25,980-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 01:10:26,400-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 01:10:35,990-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 01:10:42,662-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 01:10:46,003-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:10:56,016-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:11:02,874-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:11:06,029-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:11:16,038-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:11:19,980-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:11:26,048-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:11:36,056-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:11:39,348-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:11:46,065-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:11:56,074-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 01:11:57,298-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 01:12:06,085-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:12:15,822-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:12:16,095-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:12:26,104-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:12:34,615-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:12:36,114-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:12:46,126-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:12:52,296-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:12:56,136-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:13:06,145-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 01:13:11,934-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 01:13:16,156-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:13:26,164-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:13:28,771-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:13:36,173-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:13:46,181-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:13:49,252-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:13:56,189-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 01:14:05,245-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 01:14:06,197-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:14:16,207-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:14:26,215-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:14:26,571-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:14:36,225-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:14:41,718-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:14:46,236-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:14:56,248-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:15:03,888-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:15:06,258-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:15:16,267-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:15:18,193-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:15:26,277-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:15:36,290-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:15:41,207-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:15:46,299-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:15:54,666-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:15:56,307-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:16:06,317-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:16:16,329-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 01:16:18,525-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 01:16:26,340-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 01:16:31,141-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 01:16:36,350-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:16:46,360-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:16:55,843-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:16:56,369-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:17:06,379-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 01:17:07,615-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 01:17:16,389-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:17:26,398-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-2013-11-08 01:17:33,162-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-No streams
-2013-11-08 01:17:36,407-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:17:44,089-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:17:46,417-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:17:56,426-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:18:06,436-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:18:10,479-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:18:16,447-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 9ms
-2013-11-08 01:18:20,570-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 9ms
-No streams
-2013-11-08 01:18:26,456-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:18:36,465-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:18:46,474-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:18:47,797-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:18:56,483-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:18:57,036-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:19:06,492-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:19:16,502-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:19:25,115-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:19:26,510-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:19:33,511-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:19:36,519-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:19:46,527-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:19:56,536-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 5ms
-2013-11-08 01:20:02,436-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 5ms
-No streams
-2013-11-08 01:20:06,543-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:20:09,984-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:20:16,551-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:20:26,559-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:20:36,568-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:20:39,750-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:20:46,458-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:20:46,576-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:20:56,584-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:21:06,592-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:21:16,600-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:21:17,069-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:21:22,933-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:21:26,608-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:21:36,617-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:21:46,627-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:21:54,387-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:21:56,637-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:21:59,407-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:22:06,650-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:22:16,657-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:22:26,664-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:22:31,705-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:22:35,880-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:22:36,684-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:22:46,690-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:22:56,704-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:23:06,715-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:23:09,023-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 01:23:12,355-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 01:23:16,724-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:23:26,733-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:23:36,744-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:23:46,342-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:23:46,756-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 01:23:48,829-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 01:23:56,768-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:24:06,778-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:24:16,786-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 01:24:23,660-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:24:25,302-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:24:26,796-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:24:36,814-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:24:46,828-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:24:56,846-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:25:00,978-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:25:01,777-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:25:06,860-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:25:16,870-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:25:26,879-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:25:36,887-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:25:38,250-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:25:38,295-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:25:46,895-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:25:56,903-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:26:06,911-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:26:14,725-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:26:15,613-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:26:16,919-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:26:26,930-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:26:36,942-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:26:46,951-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:26:51,199-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:26:52,932-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:26:56,970-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:27:06,980-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:27:16,988-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:27:27,001-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:27:27,672-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:27:30,249-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:27:37,020-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:27:47,037-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:27:57,044-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:28:04,146-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:28:07,053-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 01:28:07,568-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 01:28:17,063-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:28:27,077-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:28:37,086-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:28:40,621-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:28:44,886-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:28:47,095-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:28:57,105-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:29:07,114-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:29:17,094-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:29:17,128-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:29:22,204-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:29:27,138-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:29:37,147-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:29:47,155-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:29:53,569-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:29:57,165-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:29:59,522-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:30:07,173-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:30:17,184-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:30:27,204-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:30:30,043-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:30:36,839-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:30:37,223-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:30:47,243-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:30:57,246-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 01:31:06,517-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 01:31:07,263-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:31:14,158-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:31:17,282-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:31:27,302-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:31:37,319-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:31:42,990-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:31:47,329-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:31:51,476-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:31:57,338-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:32:07,346-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:32:17,355-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:32:19,465-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:32:27,364-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:32:28,793-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:32:37,372-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:32:47,380-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:32:55,938-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:32:57,388-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:33:06,112-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:33:07,396-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:33:17,404-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:33:27,414-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:33:32,412-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:33:37,414-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:33:43,430-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:33:47,423-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:33:57,432-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:34:07,442-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:34:08,887-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:34:17,451-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:34:20,748-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:34:27,465-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:34:37,479-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:34:45,360-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:34:47,489-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:34:57,499-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:34:58,066-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:35:07,508-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:35:17,517-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:35:21,835-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:35:27,528-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:35:35,384-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:35:37,538-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:35:47,549-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:35:57,561-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:35:58,309-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:36:07,571-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 6ms
-2013-11-08 01:36:12,706-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 6ms
-No streams
-2013-11-08 01:36:17,580-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:36:27,590-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:36:34,783-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:36:37,600-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:36:47,609-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:36:50,020-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:36:57,619-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:37:07,629-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 01:37:11,257-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 01:37:17,639-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:37:27,338-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:37:27,648-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:37:37,657-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:37:47,668-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:37:47,730-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:37:57,678-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 01:38:04,655-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 01:38:07,688-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:38:17,697-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:38:24,205-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:38:27,707-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:38:37,718-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:38:41,974-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:38:47,728-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:38:57,738-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:39:00,679-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:39:07,748-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:39:17,757-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:39:19,292-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:39:27,767-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:39:37,153-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:39:37,780-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:39:47,789-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:39:56,610-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:39:57,801-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:40:07,812-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:40:13,627-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:40:17,823-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:40:27,836-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:40:33,928-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:40:37,850-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:40:47,859-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:40:50,100-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:40:57,872-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:41:07,885-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:41:11,246-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:41:17,895-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:41:26,575-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:41:27,911-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:41:37,919-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:41:47,928-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 01:41:48,563-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 01:41:57,936-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:42:03,048-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:42:07,944-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:42:17,952-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:42:25,881-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:42:27,961-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:42:37,969-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 01:42:39,522-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 01:42:47,977-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:42:57,986-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 3ms
-2013-11-08 01:43:03,201-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 3ms
-No streams
-2013-11-08 01:43:07,994-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:43:15,996-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:43:18,002-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:43:28,010-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:43:38,018-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:43:40,517-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:43:48,026-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:43:52,471-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:43:58,035-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:44:08,043-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:44:17,836-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:44:18,051-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:44:28,059-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:44:28,944-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:44:38,068-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:44:48,077-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 01:44:55,153-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 01:44:58,085-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:45:05,418-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:45:08,094-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:45:18,102-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:45:28,110-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:45:32,472-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:45:38,118-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:45:41,892-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:45:48,126-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:45:58,135-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:46:08,143-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 01:46:09,790-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 01:46:18,150-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:46:18,367-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:46:28,158-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:46:38,166-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 01:46:47,108-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 01:46:48,174-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:46:54,840-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:46:58,182-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:47:08,190-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:47:18,197-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:47:24,427-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:47:28,205-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:47:31,314-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:47:38,213-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:47:48,220-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:47:58,228-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:48:01,744-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:48:07,788-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:48:08,235-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:48:18,243-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:48:28,251-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:48:38,259-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 01:48:39,062-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 01:48:44,262-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 01:48:48,267-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:48:58,275-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:49:08,283-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:49:16,380-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:49:18,291-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:49:20,736-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:49:28,299-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:49:38,307-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:49:48,315-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:49:53,699-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 01:49:57,210-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 01:49:58,323-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:50:08,331-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:50:18,340-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:50:28,348-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:50:31,018-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:50:33,684-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:50:38,356-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:50:48,364-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:50:58,372-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 01:51:08,335-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 01:51:08,380-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 01:51:10,158-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 01:51:18,388-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:51:28,396-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:51:38,404-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:51:45,654-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:51:46,632-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:51:48,412-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:51:58,420-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:52:08,428-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:52:18,435-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 01:52:22,971-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 01:52:23,106-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 01:52:28,443-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:52:38,450-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:52:48,458-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:52:58,466-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:52:59,581-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:53:00,289-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:53:08,474-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:53:18,482-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:53:28,490-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 01:53:36,055-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:53:37,608-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:53:38,498-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:53:48,506-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:53:58,513-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:54:08,521-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:54:12,530-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 01:54:14,925-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 01:54:18,529-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:54:28,537-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:54:38,545-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:54:48,554-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 01:54:49,004-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:54:52,244-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:54:58,562-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:55:08,570-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:55:18,578-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:55:25,479-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:55:28,586-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:55:29,563-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:55:38,594-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:55:48,602-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:55:58,610-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:56:01,954-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:56:06,880-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:56:08,618-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:56:18,626-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:56:28,634-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-08 01:56:38,426-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-08 01:56:38,642-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:56:44,199-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:56:48,650-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:56:58,658-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:57:08,665-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:57:14,901-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:57:18,673-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 01:57:21,516-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 01:57:28,681-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:57:38,689-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:57:48,697-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:57:51,376-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:57:58,704-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 01:57:58,834-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 01:58:08,712-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:58:18,720-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:58:27,850-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:58:28,728-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 01:58:36,152-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 01:58:38,736-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:58:48,743-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:58:58,752-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 01:59:04,325-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 01:59:08,760-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:59:13,470-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:59:18,768-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:59:28,776-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 01:59:38,784-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 01:59:40,798-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 01:59:48,792-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 01:59:50,788-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 01:59:58,801-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:00:08,809-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 02:00:17,272-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 02:00:18,817-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 02:00:28,106-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 02:00:28,825-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:00:38,834-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:00:48,842-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:00:53,746-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:00:58,850-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 02:01:05,425-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 02:01:08,858-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:01:18,866-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:01:28,875-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 02:01:30,221-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 02:01:38,883-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 02:01:42,742-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 02:01:48,891-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:01:58,900-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-2013-11-08 02:02:06,696-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-No streams
-2013-11-08 02:02:08,908-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:02:18,916-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:02:20,061-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:02:28,925-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:02:38,933-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 02:02:43,168-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 02:02:48,941-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 02:02:57,379-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 02:02:58,949-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:03:08,957-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:03:18,965-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:03:19,642-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:03:28,973-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:03:34,697-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:03:38,981-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:03:48,989-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 02:03:56,117-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 02:03:58,997-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:04:09,006-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 02:04:12,016-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 02:04:19,014-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:04:29,022-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:04:32,590-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:04:39,030-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:04:49,039-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 02:04:49,333-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 02:04:59,047-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:05:09,055-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 02:05:09,064-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 02:05:19,063-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 02:05:26,652-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 02:05:29,071-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:05:39,080-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:05:45,538-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:05:49,088-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:05:59,096-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:06:03,969-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:06:09,104-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:06:19,113-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 02:06:22,012-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 02:06:29,121-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:06:39,129-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 12ms
-2013-11-08 02:06:41,298-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 12ms
-No streams
-2013-11-08 02:06:49,137-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:06:58,486-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:06:59,146-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:07:09,153-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 02:07:18,605-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 02:07:19,161-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:07:29,169-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:07:34,960-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:07:39,177-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:07:49,185-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:07:55,923-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:07:59,193-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:08:09,201-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:08:11,434-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:08:19,209-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:08:29,217-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 02:08:33,242-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 02:08:39,225-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 02:08:47,908-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 02:08:49,233-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:08:59,241-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:09:09,249-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 02:09:10,559-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 02:09:19,257-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:09:24,382-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:09:29,265-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:09:39,273-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:09:47,878-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:09:49,281-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:09:59,290-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 02:10:00,856-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 02:10:09,298-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:10:19,306-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 02:10:25,197-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 02:10:29,314-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:10:37,330-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:10:39,322-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:10:49,330-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:10:59,339-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:11:02,514-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:11:09,347-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:11:13,804-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:11:19,355-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:11:29,363-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:11:39,371-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 02:11:39,832-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 02:11:49,379-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:11:50,278-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:11:59,387-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:12:09,395-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:12:17,150-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:12:19,404-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:12:26,752-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:12:29,412-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:12:39,419-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:12:49,427-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 02:12:54,469-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 02:12:59,435-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 02:13:03,227-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 02:13:09,443-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:13:19,451-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:13:29,459-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:13:31,786-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:13:39,466-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:13:39,700-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:13:49,474-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:13:59,482-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 02:14:09,105-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 02:14:09,490-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 02:14:16,175-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 02:14:19,498-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:14:29,506-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:14:39,513-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 02:14:46,422-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 02:14:49,521-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:14:52,648-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:14:59,529-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:15:09,537-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:15:19,545-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:15:23,741-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 02:15:29,123-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 02:15:29,553-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:15:39,560-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:15:49,568-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:15:59,576-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 02:16:01,060-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:16:05,596-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:16:09,584-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:16:19,592-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:16:29,600-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:16:38,377-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:16:39,607-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 02:16:42,071-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 02:16:49,615-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:16:59,623-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:17:09,631-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 02:17:15,696-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:17:18,544-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:17:19,639-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:17:29,647-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:17:39,655-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:17:49,663-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:17:53,013-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:17:55,018-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:17:59,671-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:18:09,679-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:18:19,687-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:18:29,696-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:18:30,331-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:18:31,492-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:18:39,704-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:18:49,712-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:18:59,720-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 02:19:07,650-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:19:07,966-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:19:09,728-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:19:19,737-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:19:29,745-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:19:39,753-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:19:44,440-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:19:44,967-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:19:49,761-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:19:59,769-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:20:09,778-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:20:19,786-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:20:20,914-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 02:20:22,286-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 02:20:29,794-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:20:39,803-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:20:49,811-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:20:57,389-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:20:59,603-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:20:59,819-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:21:09,827-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:21:19,835-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:21:29,843-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:21:33,863-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:21:36,922-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:21:39,851-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:21:49,859-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:21:59,866-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:22:09,874-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 02:22:10,337-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:22:14,240-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:22:19,882-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:22:29,890-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:22:39,897-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 02:22:46,812-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 02:22:49,905-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:22:51,558-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:22:59,912-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:23:09,920-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:23:19,928-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 02:23:23,287-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:23:28,876-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:23:29,937-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:23:39,945-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:23:49,952-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:23:59,760-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:23:59,960-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 4ms
-2013-11-08 02:24:06,197-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 4ms
-No streams
-2013-11-08 02:24:09,968-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:24:19,976-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:24:29,984-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 02:24:36,235-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 02:24:39,991-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:24:43,512-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:24:49,999-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:25:00,007-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:25:10,015-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:25:12,708-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:25:20,022-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:25:20,830-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:25:30,030-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:25:40,038-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 02:25:49,183-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 02:25:50,046-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:25:58,149-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:26:00,054-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:26:10,062-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:26:20,070-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 02:26:25,656-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 02:26:30,077-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 02:26:35,467-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 02:26:40,085-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:26:50,093-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:27:00,101-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 02:27:02,131-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 02:27:10,109-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:27:12,786-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:27:20,117-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:27:30,125-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:27:38,604-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:27:40,133-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 02:27:50,105-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 02:27:50,141-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:28:00,148-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:28:10,156-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:28:15,078-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:28:20,164-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:28:27,422-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:28:30,172-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:28:40,180-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:28:50,188-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:28:51,552-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:29:00,196-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 02:29:04,741-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 02:29:10,204-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:29:20,212-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 02:29:28,027-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 02:29:30,220-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:29:40,228-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:29:42,058-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:29:50,237-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:30:00,245-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 02:30:04,500-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 02:30:10,253-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 02:30:19,377-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 02:30:20,261-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:30:30,269-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:30:40,277-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 02:30:40,976-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 02:30:50,285-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:30:56,694-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:31:00,293-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:31:10,301-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 02:31:17,450-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 02:31:20,309-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:31:30,318-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-08 02:31:34,012-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-08 02:31:40,326-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:31:50,334-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:31:53,923-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:32:00,342-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:32:10,350-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:32:11,331-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:32:20,359-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:32:30,367-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:32:30,397-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:32:40,375-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:32:48,649-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:32:50,383-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:33:00,391-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:33:06,871-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:33:10,399-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:33:20,407-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 02:33:25,967-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 02:33:30,416-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:33:40,424-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:33:43,345-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:33:50,432-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:34:00,440-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:34:03,285-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:34:10,448-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 02:34:19,818-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 02:34:20,456-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:34:30,464-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:34:40,472-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 02:34:40,604-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 02:34:50,480-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:34:56,293-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:35:00,488-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:35:10,496-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:35:17,921-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:35:20,504-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:35:30,512-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:35:32,767-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:35:40,521-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:35:50,529-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 02:35:55,240-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 02:36:00,537-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:36:09,241-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:36:10,545-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:36:20,553-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:36:30,561-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:36:32,557-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:36:40,569-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 02:36:45,714-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 02:36:50,577-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:37:00,585-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:37:09,876-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:37:10,593-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:37:20,601-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:37:22,189-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:37:30,609-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:37:40,617-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 02:37:47,195-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 02:37:50,625-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:37:58,664-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:38:00,632-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:38:10,640-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:38:20,648-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:38:24,512-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:38:30,656-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:38:35,138-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:38:40,664-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:38:50,671-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:39:00,680-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:39:01,830-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:39:10,688-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 02:39:11,612-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 02:39:20,696-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:39:30,703-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:39:39,148-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:39:40,711-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:39:48,086-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:39:50,718-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:40:00,727-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:40:10,734-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 02:40:16,467-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 02:40:20,742-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 02:40:24,561-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 02:40:30,749-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:40:40,757-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:40:50,765-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:40:53,784-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:41:00,773-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 02:41:01,034-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 02:41:10,781-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:41:20,789-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:41:30,796-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:41:31,103-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 02:41:37,509-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 02:41:40,804-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:41:50,812-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:42:00,820-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 02:42:08,422-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 02:42:10,827-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:42:13,982-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:42:20,835-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:42:30,843-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:42:40,851-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:42:45,739-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 02:42:50,457-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 02:42:50,858-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:43:00,866-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:43:10,874-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:43:20,882-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 02:43:23,058-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 02:43:26,930-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 02:43:30,890-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:43:40,897-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:43:50,905-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:44:00,375-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:44:00,913-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 02:44:03,405-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 02:44:10,921-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:44:20,929-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:44:30,936-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 02:44:37,694-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:44:39,878-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:44:40,944-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:44:50,952-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:45:00,960-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:45:10,968-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 02:45:15,012-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:45:16,352-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:45:20,976-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:45:30,983-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:45:40,991-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:45:50,999-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:45:52,329-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:45:52,826-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:46:01,007-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:46:11,015-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:46:21,023-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 02:46:29,301-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 02:46:29,648-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 02:46:31,031-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:46:41,039-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:46:51,046-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:47:01,054-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 02:47:05,774-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:47:06,965-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:47:11,062-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:47:21,070-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:47:31,078-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:47:41,085-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:47:42,248-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:47:44,283-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:47:51,093-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:48:01,101-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:48:11,109-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 02:48:18,722-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 02:48:21,117-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 02:48:21,602-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 02:48:31,125-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:48:41,132-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:48:51,140-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:48:55,196-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:48:58,919-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:49:01,148-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:49:11,155-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:49:21,164-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:49:31,171-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 02:49:31,669-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 02:49:36,238-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 02:49:41,179-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:49:51,186-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:50:01,194-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:50:08,144-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:50:11,202-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:50:13,555-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:50:21,210-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:50:31,218-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:50:41,225-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 02:50:44,619-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 02:50:50,874-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 02:50:51,233-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:51:01,241-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:51:11,249-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:51:21,093-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:51:21,256-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 02:51:28,192-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 02:51:31,264-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:51:41,272-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:51:51,279-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 02:51:57,566-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 02:52:01,287-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:52:05,509-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:52:11,295-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:52:21,303-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:52:31,310-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 02:52:34,042-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 02:52:41,318-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:52:42,827-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:52:51,326-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:53:01,334-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 02:53:10,515-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 02:53:11,342-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:53:20,145-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:53:21,349-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:53:31,357-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:53:41,365-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 02:53:46,989-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 02:53:51,373-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 02:53:57,464-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 02:54:01,380-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:54:11,388-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:54:21,397-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 02:54:23,464-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 02:54:31,404-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:54:34,781-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:54:41,412-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:54:51,419-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:54:59,938-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:55:01,427-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:55:11,435-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:55:12,099-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:55:21,442-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:55:31,450-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 02:55:36,413-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 02:55:41,458-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:55:49,417-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:55:51,466-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:56:01,474-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:56:11,481-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:56:12,887-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:56:21,489-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:56:26,735-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:56:31,497-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:56:41,505-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 02:56:49,361-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 02:56:51,513-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:57:01,521-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:57:04,053-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:57:11,529-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:57:21,537-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:57:25,835-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:57:31,544-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:57:41,371-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:57:41,552-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:57:51,560-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:58:01,568-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 02:58:02,308-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 02:58:11,576-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 02:58:18,689-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 02:58:21,584-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:58:31,592-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:58:38,783-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:58:41,600-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:58:51,608-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:58:56,007-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:59:01,616-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:59:11,624-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 02:59:15,256-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 02:59:21,632-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:59:31,641-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 02:59:33,325-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 02:59:41,649-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 02:59:51,657-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 02:59:51,731-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 03:00:01,665-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 03:00:10,644-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 03:00:11,674-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:00:21,683-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:00:28,206-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:00:31,691-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:00:41,699-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:00:47,961-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:00:51,708-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:01:01,716-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:01:04,680-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:01:11,724-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:01:21,732-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:01:25,279-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:01:31,741-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 03:01:41,153-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 03:01:41,749-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:01:51,757-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:02:01,765-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:02:02,597-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:02:11,773-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 03:02:17,628-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 03:02:21,782-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:02:31,790-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:02:39,915-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:02:41,791-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:02:51,799-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 03:02:54,102-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 03:03:01,808-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:03:11,816-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 03:03:17,234-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 03:03:21,824-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:03:30,577-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:03:31,832-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:03:41,841-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:03:51,849-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:03:54,551-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:04:01,857-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:04:07,051-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:04:11,865-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:04:21,874-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 03:04:31,870-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 03:04:31,882-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:04:41,890-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:04:43,525-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:04:51,899-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:05:01,907-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 03:05:09,188-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 03:05:11,915-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:05:19,999-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:05:21,923-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:05:31,931-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:05:41,940-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:05:46,505-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:05:51,948-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:05:56,473-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:06:01,956-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:06:11,964-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:06:21,973-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:06:23,823-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:06:31,981-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:06:32,947-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:06:41,989-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:06:51,998-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:07:01,141-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:07:02,006-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:07:09,421-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:07:12,014-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:07:22,023-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:07:32,031-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 03:07:38,460-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 03:07:42,039-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:07:45,895-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:07:52,047-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:08:02,056-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:08:12,064-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:08:15,777-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:08:22,072-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 03:08:22,369-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 03:08:32,081-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:08:42,089-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:08:52,097-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:08:53,096-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 03:08:58,844-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 03:09:02,105-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:09:12,113-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:09:22,122-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 03:09:30,415-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 03:09:32,129-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:09:35,317-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:09:42,137-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:09:52,145-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:10:02,153-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:10:07,732-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 03:10:11,792-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 03:10:12,161-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:10:22,169-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:10:32,177-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:10:42,185-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 03:10:45,051-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 03:10:48,265-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 03:10:52,193-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:11:02,201-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:11:12,209-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:11:22,217-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:11:22,368-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:11:24,739-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:11:32,224-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:11:42,232-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:11:52,240-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:11:59,687-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 03:12:01,213-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 03:12:02,248-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:12:12,256-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:12:22,264-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:12:32,272-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 03:12:37,006-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:12:37,687-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:12:42,281-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:12:52,289-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:13:02,298-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:13:12,306-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 03:13:14,161-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:13:14,323-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:13:22,315-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:13:32,325-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:13:42,333-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:13:50,635-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 03:13:51,642-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 03:13:52,342-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:14:02,350-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:14:12,358-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:14:22,367-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 03:14:27,108-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:14:28,959-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:14:32,375-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:14:42,383-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:14:52,392-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:15:02,400-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:15:03,583-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:15:06,277-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:15:12,409-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:15:22,418-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:15:32,426-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 03:15:40,058-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 03:15:42,435-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:15:43,595-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:15:52,443-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:16:02,452-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:16:12,460-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:16:16,532-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:16:20,913-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:16:22,469-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:16:32,477-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:16:42,486-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:16:52,494-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 03:16:53,005-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 03:16:58,232-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 03:17:02,502-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:17:12,511-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:17:22,519-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:17:29,480-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:17:32,527-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:17:35,549-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:17:42,535-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:17:52,544-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:18:02,552-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 03:18:05,953-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 03:18:12,560-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:18:12,868-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:18:22,569-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:18:32,578-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:18:42,428-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:18:42,586-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:18:50,186-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:18:52,594-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:19:02,602-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:19:12,611-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 03:19:18,902-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 03:19:22,619-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:19:27,504-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:19:32,627-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:19:42,635-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:19:52,644-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:19:55,377-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:20:02,652-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:20:04,822-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:20:12,661-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:20:22,669-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 03:20:31,852-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 03:20:32,677-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:20:42,140-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:20:42,685-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:20:52,693-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:21:02,701-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 03:21:08,325-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 03:21:12,710-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 03:21:19,459-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 03:21:22,718-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:21:32,726-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:21:42,734-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:21:44,799-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:21:52,744-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:21:56,776-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:22:02,752-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:22:12,760-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:22:21,273-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:22:22,769-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:22:32,777-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:22:34,094-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:22:42,786-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:22:52,794-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:22:57,747-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:23:02,802-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:23:11,412-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:23:12,810-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:23:22,819-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:23:32,827-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 03:23:34,221-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 03:23:42,835-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 03:23:48,731-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 03:23:52,844-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:24:02,852-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:24:10,695-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:24:12,860-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:24:22,868-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:24:26,049-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:24:32,876-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:24:42,885-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 03:24:47,169-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 03:24:52,892-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:25:02,900-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:25:03,367-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:25:12,909-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:25:22,917-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:25:23,643-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:25:32,925-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 03:25:40,686-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 03:25:42,932-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:25:52,940-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 03:26:00,117-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 03:26:02,949-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:26:12,957-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:26:18,003-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:26:22,965-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:26:32,972-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:26:36,591-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:26:42,981-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:26:52,990-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:26:55,321-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:27:02,998-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:27:13,006-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:27:13,065-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:27:23,014-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 03:27:32,640-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 03:27:33,022-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:27:43,030-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:27:49,540-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:27:53,038-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:28:03,046-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:28:09,958-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:28:13,054-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:28:23,062-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 03:28:26,014-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 03:28:33,073-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:28:43,084-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 03:28:47,277-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 03:28:53,093-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:29:02,487-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:29:03,103-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:29:13,112-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:29:23,122-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:29:24,595-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:29:33,131-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:29:38,961-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:29:43,142-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:29:53,147-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:30:01,914-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:30:03,153-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:30:13,155-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:30:15,435-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:30:23,169-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:30:33,180-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 03:30:39,233-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 03:30:43,189-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:30:51,909-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:30:53,201-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:31:03,213-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:31:13,226-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:31:16,551-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:31:23,239-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:31:28,383-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:31:33,250-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:31:43,261-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:31:53,272-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:31:53,869-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:32:03,283-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 03:32:04,858-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 03:32:13,293-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:32:23,301-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 03:32:31,188-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 03:32:33,310-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 03:32:41,332-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 03:32:43,319-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:32:53,327-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:33:03,335-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 03:33:08,506-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 03:33:13,345-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 03:33:17,806-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 03:33:23,355-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:33:33,373-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:33:43,384-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 03:33:45,824-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 03:33:53,395-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:33:54,279-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:34:03,407-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:34:13,415-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 03:34:23,142-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 03:34:23,423-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:34:30,753-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:34:33,433-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:34:43,442-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:34:53,451-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 03:35:00,460-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 03:35:03,461-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 03:35:07,228-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 03:35:13,469-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:35:23,477-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:35:33,485-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 03:35:37,778-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 03:35:43,493-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:35:43,701-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:35:53,502-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:36:03,511-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:36:13,520-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 03:36:15,096-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 03:36:20,176-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 03:36:23,530-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:36:33,540-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:36:43,553-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 03:36:52,414-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 03:36:53,564-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:36:56,649-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:37:03,574-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:37:13,584-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:37:23,593-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 03:37:29,732-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 03:37:33,124-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 03:37:33,602-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:37:43,612-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:37:53,622-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:38:03,632-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:38:07,049-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:38:09,597-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:38:13,641-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:38:23,650-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:38:33,659-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:38:43,668-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 03:38:44,368-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:38:46,071-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:38:53,679-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:39:03,688-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:39:13,697-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 03:39:21,686-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 03:39:22,546-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 03:39:23,708-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:39:33,717-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:39:43,726-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:39:53,735-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:39:59,003-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:39:59,019-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:40:03,749-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:40:13,761-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:40:23,771-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:40:33,780-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 03:40:35,494-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:40:36,321-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:40:43,790-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:40:53,800-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:41:03,810-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 03:41:11,968-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 03:41:13,639-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 03:41:13,818-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:41:23,826-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:41:33,834-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:41:43,842-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:41:48,441-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:41:50,958-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:41:53,851-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:42:03,859-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:42:13,867-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:42:23,875-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:42:24,915-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:42:28,276-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:42:33,883-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:42:43,892-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:42:53,900-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 03:43:01,389-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 03:43:03,908-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:43:05,594-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:43:13,916-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:43:23,924-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:43:33,932-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:43:37,864-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:43:42,912-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:43:43,940-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:43:53,948-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:44:03,956-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:44:13,964-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:44:14,338-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 03:44:20,231-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 03:44:23,973-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:44:33,981-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:44:43,989-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:44:50,812-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:44:53,997-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:44:57,548-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:45:04,005-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:45:14,013-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:45:24,021-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 03:45:27,287-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 03:45:34,029-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 03:45:34,867-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 03:45:44,037-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:45:54,045-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:46:03,760-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:46:04,054-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:46:12,184-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:46:14,061-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:46:24,069-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:46:34,078-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:46:40,234-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:46:44,086-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 03:46:49,504-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 03:46:54,094-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:47:04,102-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:47:14,110-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:47:16,708-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:47:24,118-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 03:47:26,821-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 03:47:34,126-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:47:44,135-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:47:53,182-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:47:54,143-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 03:48:04,138-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 03:48:04,151-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:48:14,159-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:48:24,167-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 03:48:29,655-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 03:48:34,175-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 03:48:41,457-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 03:48:44,184-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:48:54,191-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:49:04,200-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:49:06,130-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:49:14,208-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 03:49:18,774-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 03:49:24,216-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:49:34,224-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 03:49:42,603-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 03:49:44,232-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:49:54,240-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:49:56,092-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:50:04,248-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:50:14,256-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:50:19,078-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:50:24,264-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:50:33,410-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:50:34,272-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:50:44,280-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:50:54,288-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 03:50:55,553-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 03:51:04,296-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 03:51:10,729-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 03:51:14,304-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:51:24,312-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:51:32,027-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:51:34,321-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:51:44,329-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:51:48,046-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:51:54,337-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:52:04,345-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:52:08,501-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:52:14,353-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:52:24,361-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:52:25,365-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:52:34,369-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:52:44,377-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:52:44,975-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:52:54,385-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:53:02,683-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:53:04,394-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:53:14,402-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 03:53:21,450-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 03:53:24,410-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:53:34,418-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 03:53:40,001-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 03:53:44,426-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:53:54,434-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:53:57,923-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:54:04,442-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:54:14,450-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 03:54:17,320-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 03:54:24,458-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:54:34,397-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:54:34,466-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:54:44,474-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:54:54,483-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:54:54,637-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:55:04,491-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:55:10,871-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:55:14,499-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:55:24,507-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 03:55:31,956-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 03:55:34,515-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:55:44,523-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:55:47,345-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:55:54,532-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:56:04,540-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:56:09,273-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:56:14,548-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:56:23,819-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:56:24,556-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:56:34,564-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:56:44,572-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 03:56:46,592-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 03:56:54,580-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:57:00,293-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:57:04,588-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:57:14,596-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:57:23,909-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:57:24,604-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:57:34,612-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 03:57:36,767-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 03:57:44,620-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:57:54,628-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 03:58:01,227-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 03:58:04,636-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:58:13,241-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:58:14,644-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:58:24,652-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:58:34,660-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:58:38,545-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:58:44,668-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 03:58:49,715-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 03:58:54,676-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:59:04,684-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:59:14,692-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 03:59:15,863-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 03:59:24,700-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 03:59:26,190-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 03:59:34,708-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 03:59:44,716-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 03:59:53,182-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 03:59:54,724-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 04:00:02,665-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 04:00:04,732-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:00:14,740-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:00:24,749-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:00:30,499-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:00:34,757-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:00:39,138-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:00:44,765-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:00:54,774-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:01:04,782-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 04:01:07,818-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 04:01:14,790-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:01:15,612-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:01:24,799-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:01:34,807-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:01:44,815-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:01:45,135-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:01:52,086-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:01:54,823-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:02:04,831-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:02:14,839-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 04:02:22,453-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 04:02:24,847-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:02:28,560-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:02:34,855-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:02:44,863-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:02:54,871-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:02:59,771-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:03:04,879-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 04:03:05,033-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 04:03:14,887-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:03:24,895-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:03:34,902-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 04:03:37,089-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:03:41,508-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:03:44,909-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:03:54,917-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:04:04,925-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:04:14,407-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:04:14,933-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:04:17,982-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:04:24,941-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:04:34,949-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:04:44,957-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:04:51,725-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:04:54,456-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:04:54,965-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:05:04,973-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:05:14,981-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:05:24,989-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:05:29,043-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 04:05:30,930-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 04:05:34,997-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:05:45,005-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:05:55,013-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:06:05,021-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:06:06,361-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:06:07,404-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:06:15,029-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:06:25,037-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:06:35,045-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 04:06:43,679-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 04:06:43,878-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 04:06:45,053-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:06:55,061-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:07:05,069-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:07:15,077-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:07:20,352-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:07:20,998-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:07:25,086-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:07:35,094-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:07:45,102-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:07:55,110-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:07:56,827-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 04:07:58,316-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 04:08:05,118-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:08:15,126-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:08:25,134-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:08:33,300-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:08:35,142-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 04:08:35,635-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 04:08:45,151-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:08:55,159-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:09:05,167-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 04:09:09,775-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-2013-11-08 04:09:12,954-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-No streams
-2013-11-08 04:09:15,175-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:09:25,183-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:09:35,191-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:09:45,199-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 04:09:46,248-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 04:09:50,270-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 04:09:55,207-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:10:05,216-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:10:15,224-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 04:10:22,723-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 04:10:25,232-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:10:27,588-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:10:35,240-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:10:45,249-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:10:55,257-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:10:59,196-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 04:11:04,906-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 04:11:05,265-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:11:15,273-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:11:25,281-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:11:35,289-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:11:35,670-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 04:11:42,225-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 04:11:45,298-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:11:55,306-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:12:05,314-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:12:12,144-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:12:15,322-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 04:12:19,542-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 04:12:25,330-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:12:35,338-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:12:45,346-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:12:48,618-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:12:55,354-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 04:12:56,861-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 04:13:05,363-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:13:15,371-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 04:13:25,091-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 04:13:25,379-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 04:13:34,178-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 04:13:35,387-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:13:45,395-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:13:55,403-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:14:01,566-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:14:05,411-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 04:14:11,497-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 04:14:15,420-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:14:25,428-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:14:35,436-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 04:14:38,040-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 04:14:45,444-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 04:14:48,814-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 04:14:55,452-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:15:05,460-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:15:14,514-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:15:15,469-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:15:25,477-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 04:15:26,132-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 04:15:35,485-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:15:45,493-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 04:15:50,988-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 04:15:55,502-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 04:16:03,451-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 04:16:05,510-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:16:15,518-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:16:25,527-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:16:27,462-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:16:35,535-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 04:16:40,768-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 04:16:45,543-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:16:55,551-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 04:17:03,936-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 04:17:05,560-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:17:15,568-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 04:17:18,087-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 04:17:25,576-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:17:35,584-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:17:40,410-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:17:45,592-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:17:55,404-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:17:55,600-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:18:05,608-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:18:15,616-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 04:18:16,884-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 04:18:25,624-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 04:18:32,723-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 04:18:35,632-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:18:45,640-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:18:53,358-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:18:55,648-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:19:05,656-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:19:10,041-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:19:15,665-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:19:25,672-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:19:29,832-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:19:35,680-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:19:45,689-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 04:19:47,359-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 04:19:55,697-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:20:05,705-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:20:06,306-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:20:15,713-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 04:20:24,678-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 04:20:25,722-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:20:35,730-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 04:20:42,781-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 04:20:45,738-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:20:55,746-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 04:21:01,995-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 04:21:05,755-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:21:15,763-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:21:19,254-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:21:25,771-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:21:35,779-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 04:21:39,314-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 04:21:45,788-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:21:55,728-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:21:55,796-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:22:05,804-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:22:15,812-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:22:16,631-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:22:25,820-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:22:32,202-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:22:35,829-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:22:45,837-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 04:22:53,950-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 04:22:55,845-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:23:05,853-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:23:08,676-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:23:15,861-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:23:25,870-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:23:31,268-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:23:35,878-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:23:45,150-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:23:45,886-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:23:55,894-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:24:05,903-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 04:24:08,586-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 04:24:15,910-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:24:21,624-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:24:25,918-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:24:35,927-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:24:45,904-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:24:45,935-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:24:55,943-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 04:24:58,098-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 04:25:05,951-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:25:15,959-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:25:23,222-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:25:25,967-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 04:25:34,573-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 04:25:35,975-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:25:45,984-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:25:55,992-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 04:26:00,541-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 04:26:06,000-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 04:26:11,045-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 04:26:16,009-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:26:26,017-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:26:36,025-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:26:37,858-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:26:46,034-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:26:47,520-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:26:56,042-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:27:06,050-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 04:27:15,176-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 04:27:16,058-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:27:23,994-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:27:26,067-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:27:36,075-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:27:46,083-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:27:52,494-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:27:56,091-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:28:00,468-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:28:06,100-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:28:16,108-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:28:26,116-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:28:29,812-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:28:36,124-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:28:36,942-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:28:46,131-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:28:56,139-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:29:06,147-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-2013-11-08 04:29:07,132-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 04:29:13,417-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 04:29:16,155-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:29:26,163-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:29:36,171-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:29:44,448-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:29:46,179-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:29:49,890-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:29:56,187-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:30:06,195-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:30:16,203-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 04:30:21,766-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 04:30:26,212-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:30:26,364-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:30:36,220-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:30:46,228-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:30:56,236-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:30:59,084-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:31:02,838-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:31:06,244-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:31:16,253-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:31:26,261-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:31:36,269-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:31:36,402-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:31:39,312-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:31:46,278-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:31:56,286-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:32:06,294-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:32:13,720-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 04:32:15,786-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 04:32:16,302-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:32:26,310-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:32:36,319-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:32:46,327-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:32:51,038-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:32:52,260-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:32:56,335-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:33:06,343-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:33:16,351-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:33:26,360-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 04:33:28,357-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 04:33:28,733-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 04:33:36,368-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:33:46,376-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:33:56,384-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:34:05,208-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:34:05,675-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:34:06,392-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:34:16,400-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:34:26,409-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:34:36,417-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:34:41,682-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 04:34:42,993-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 04:34:46,425-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:34:56,433-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:35:06,441-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:35:16,449-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:35:18,156-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 04:35:20,312-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 04:35:26,457-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:35:36,465-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:35:46,474-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 04:35:54,631-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 04:35:56,482-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 04:35:57,630-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 04:36:06,490-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:36:16,498-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:36:26,507-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 04:36:31,104-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 04:36:34,948-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 04:36:36,515-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:36:46,523-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:36:56,532-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:37:06,540-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:37:07,578-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:37:12,265-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:37:16,548-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:37:26,557-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:37:36,565-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:37:44,052-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:37:46,573-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 04:37:49,583-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 04:37:56,581-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:38:06,590-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:38:16,598-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:38:20,526-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:38:26,606-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 04:38:26,902-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 04:38:36,614-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:38:46,622-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:38:56,630-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 04:38:57,000-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 04:39:04,219-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 04:39:06,638-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:39:16,646-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:39:26,654-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:39:33,474-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:39:36,662-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:39:41,537-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:39:46,670-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:39:56,678-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:40:06,686-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 04:40:09,948-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 04:40:16,694-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 04:40:18,855-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 04:40:26,702-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:40:36,710-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:40:46,422-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:40:46,719-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 04:40:56,174-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 04:40:56,726-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:41:06,734-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:41:16,742-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 04:41:22,896-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 04:41:26,750-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:41:33,491-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:41:36,758-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:41:46,766-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:41:56,774-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:41:59,370-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:42:06,782-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 04:42:10,810-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 04:42:16,790-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:42:26,798-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:42:35,844-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:42:36,806-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:42:46,814-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:42:48,127-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:42:56,822-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:43:06,830-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 04:43:12,318-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 04:43:16,838-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 04:43:25,446-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 04:43:26,846-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:43:36,854-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:43:46,862-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:43:48,792-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:43:56,871-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:44:02,764-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:44:06,879-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:44:16,887-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:44:25,266-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:44:26,895-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:44:36,903-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 04:44:40,082-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 04:44:46,912-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:44:56,920-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:45:01,740-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:45:06,928-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:45:16,936-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:45:17,400-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:45:26,945-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:45:36,953-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:45:38,214-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:45:46,961-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 04:45:54,718-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 04:45:56,969-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:46:06,977-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:46:14,688-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:46:16,986-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:46:26,994-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 04:46:32,037-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 04:46:37,001-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:46:47,009-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 04:46:51,162-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 04:46:57,017-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:47:07,025-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:47:09,354-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:47:17,033-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:47:27,041-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:47:27,636-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:47:37,049-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 04:47:46,673-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 04:47:47,058-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:47:57,066-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 04:48:04,110-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 04:48:07,075-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:48:17,083-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 04:48:23,990-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 04:48:27,091-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:48:37,099-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:48:40,584-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:48:47,107-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:48:57,116-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 04:49:01,308-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 04:49:07,124-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 04:49:17,058-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 04:49:17,132-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:49:27,141-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:49:37,149-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:49:38,626-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:49:47,157-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:49:53,533-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:49:57,166-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:50:07,174-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 04:50:15,944-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 04:50:17,182-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:50:27,190-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 04:50:30,008-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 04:50:37,199-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:50:47,207-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 4ms
-2013-11-08 04:50:53,265-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 4ms
-No streams
-2013-11-08 04:50:57,215-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 04:51:06,481-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 04:51:07,223-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:51:17,231-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:51:27,239-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 04:51:30,580-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 04:51:37,247-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:51:42,955-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:51:47,255-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:51:57,263-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:52:07,271-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 04:52:07,899-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 04:52:17,279-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:52:19,429-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:52:27,287-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:52:37,295-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 04:52:45,216-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 04:52:47,304-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:52:55,903-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:52:57,312-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:53:07,320-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:53:17,328-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 04:53:22,535-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 04:53:27,336-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 04:53:32,376-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 04:53:37,344-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:53:47,352-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:53:57,360-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:53:59,852-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:54:07,368-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:54:08,851-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:54:17,378-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:54:27,386-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 04:54:37,171-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 04:54:37,394-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 04:54:45,325-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 04:54:47,402-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:54:57,410-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:55:07,419-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:55:14,488-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:55:17,427-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:55:21,800-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:55:27,435-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:55:37,443-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:55:47,452-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 04:55:51,807-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 04:55:57,460-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:55:58,274-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:56:07,468-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:56:17,476-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:56:27,484-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:56:29,125-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:56:34,748-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:56:37,493-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:56:47,501-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:56:57,509-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 04:57:06,443-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 04:57:07,517-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:57:11,222-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:57:17,525-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:57:27,534-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:57:37,542-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:57:43,761-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:57:47,550-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 04:57:47,695-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 04:57:57,558-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:58:07,566-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:58:17,574-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 04:58:21,079-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:58:24,170-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:58:27,582-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:58:37,590-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:58:47,599-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:58:57,606-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 04:58:58,398-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 04:59:00,644-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 04:59:07,615-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:59:17,623-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:59:27,631-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:59:35,715-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 04:59:37,118-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 04:59:37,639-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:59:47,647-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 04:59:57,655-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:00:07,663-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:00:13,034-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:00:13,593-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:00:17,672-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:00:27,680-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:00:37,688-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:00:47,697-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:00:50,067-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:00:50,351-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:00:57,705-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:01:07,713-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:01:17,721-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 05:01:26,542-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:01:27,669-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:01:27,729-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:01:37,737-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:01:47,746-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:01:57,754-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:02:03,015-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:02:04,987-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:02:07,762-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:02:17,770-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:02:27,778-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:02:37,786-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:02:39,489-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:02:42,305-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:02:47,794-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:02:57,802-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:03:07,811-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:03:15,963-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:03:17,819-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:03:19,624-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:03:27,827-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:03:37,835-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:03:47,843-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:03:52,437-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:03:56,941-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:03:57,851-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:04:07,859-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:04:17,867-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:04:27,875-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:04:28,911-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 05:04:34,260-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 05:04:37,883-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:04:47,891-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:04:57,899-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:05:05,385-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:05:07,908-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:05:11,577-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:05:17,916-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:05:27,924-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:05:37,932-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:05:41,859-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:05:47,940-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 05:05:48,895-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 05:05:57,948-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:06:07,956-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:06:17,964-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:06:18,333-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:06:26,213-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:06:27,972-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:06:37,980-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:06:47,988-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 05:06:54,808-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 05:06:57,996-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:07:03,531-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:07:08,004-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:07:18,013-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:07:28,021-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:07:31,281-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:07:38,029-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:07:40,849-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:07:48,037-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:07:58,045-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:08:07,755-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:08:08,053-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:08:18,062-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:08:18,167-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:08:28,070-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:08:38,078-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:08:44,229-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:08:48,086-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:08:55,485-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:08:58,094-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:09:08,102-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:09:18,110-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:09:20,703-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:09:28,118-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:09:32,803-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:09:38,127-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:09:48,134-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:09:57,177-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:09:58,143-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:10:08,150-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:10:10,121-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:10:18,159-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:10:28,166-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:10:33,651-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:10:38,175-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:10:47,439-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:10:48,183-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:10:58,191-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:11:08,198-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:11:10,125-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:11:18,207-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:11:24,758-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:11:28,215-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:11:38,223-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:11:46,599-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:11:48,231-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:11:58,239-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:12:02,075-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:12:08,247-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:12:18,255-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 05:12:23,072-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 05:12:28,264-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:12:38,272-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:12:39,393-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:12:48,280-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:12:58,288-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:12:59,547-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:13:08,296-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:13:16,711-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:13:18,305-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:13:28,312-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:13:36,022-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:13:38,321-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:13:48,329-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:13:54,029-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:13:58,337-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:14:08,345-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:14:12,495-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:14:18,354-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:14:28,362-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:14:31,347-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:14:38,370-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:14:48,378-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:14:48,969-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:14:58,386-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:15:08,395-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:15:08,665-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:15:18,403-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:15:25,443-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:15:28,412-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:15:38,420-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 05:15:45,984-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 05:15:48,432-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:15:58,440-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:16:01,917-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:16:08,448-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:16:18,456-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:16:23,301-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:16:28,464-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:16:38,391-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:16:38,472-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:16:48,480-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:16:58,488-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:17:00,619-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:17:08,496-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:17:14,865-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:17:18,504-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:17:28,512-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:17:37,937-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:17:38,521-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:17:48,529-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:17:51,339-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:17:58,537-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:18:08,545-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:18:15,255-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:18:18,553-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:18:27,813-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:18:28,562-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:18:38,570-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:18:48,578-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:18:52,573-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:18:58,586-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:19:04,287-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:19:08,594-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:19:18,602-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:19:28,610-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:19:29,891-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:19:38,618-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:19:40,761-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:19:48,626-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:19:58,634-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 05:20:07,210-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 05:20:08,642-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:20:17,235-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:20:18,650-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:20:28,657-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:20:38,665-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:20:44,527-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:20:48,673-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 05:20:53,710-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 05:20:58,680-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:21:08,688-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:21:18,696-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:21:21,846-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:21:28,704-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:21:30,183-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:21:38,712-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:21:48,720-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:21:58,727-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:21:59,163-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:22:06,657-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:22:08,735-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:22:18,743-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:22:28,751-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:22:36,481-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:22:38,759-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 05:22:43,131-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 05:22:48,768-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:22:58,776-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:23:08,784-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:23:13,799-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:23:18,792-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:23:19,606-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:23:28,800-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:23:38,808-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:23:48,816-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:23:51,117-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:23:56,080-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:23:58,825-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:24:08,833-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:24:18,841-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:24:28,435-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:24:28,849-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:24:32,554-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:24:38,858-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:24:48,866-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:24:58,874-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:25:05,753-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:25:08,882-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:25:09,028-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:25:18,890-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:25:28,899-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:25:38,907-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 05:25:43,072-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:25:45,502-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:25:48,915-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:25:58,924-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:26:08,932-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:26:18,940-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:26:20,389-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:26:21,976-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:26:28,949-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:26:38,957-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:26:48,966-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:26:57,707-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:26:58,450-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:26:58,974-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:27:08,983-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:27:18,991-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:27:28,999-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 05:27:34,925-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:27:35,025-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:27:39,008-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:27:49,016-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:27:59,024-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:28:09,032-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 05:28:11,397-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:28:12,343-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:28:19,040-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:28:29,049-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:28:39,057-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:28:47,872-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:28:49,065-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 05:28:49,662-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 05:28:59,073-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:29:09,082-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:29:19,090-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:29:24,347-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:29:26,979-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:29:29,098-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:29:39,106-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:29:49,114-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:29:59,123-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:30:00,821-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 4ms
-2013-11-08 05:30:04,300-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 4ms
-No streams
-2013-11-08 05:30:09,131-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:30:19,139-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:30:29,147-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:30:37,296-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:30:39,156-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:30:41,616-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:30:49,164-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:30:59,172-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:31:09,180-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:31:13,770-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:31:18,933-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:31:19,188-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:31:29,196-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:31:39,205-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:31:49,213-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:31:50,244-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 05:31:56,252-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 05:31:59,221-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:32:09,228-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:32:19,237-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:32:26,718-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:32:29,245-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:32:33,569-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:32:39,253-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:32:49,263-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:32:59,272-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:33:03,192-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:33:09,280-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 05:33:10,888-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 05:33:19,288-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:33:29,297-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:33:39,305-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:33:39,666-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:33:48,205-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:33:49,314-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:33:59,322-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:34:09,330-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:34:16,140-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:34:19,339-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:34:25,524-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:34:29,347-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:34:39,356-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:34:49,364-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:34:52,614-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:34:59,373-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:35:02,842-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:35:09,381-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:35:19,389-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:35:29,088-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:35:29,398-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:35:39,406-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:35:40,160-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:35:49,415-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:35:59,423-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:36:05,562-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:36:09,432-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 05:36:17,479-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 05:36:19,440-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:36:29,448-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:36:39,457-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:36:42,036-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:36:49,465-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:36:54,796-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:36:59,473-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:37:09,482-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:37:18,510-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:37:19,490-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:37:29,498-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 05:37:32,115-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 05:37:39,507-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:37:49,515-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:37:54,984-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:37:59,523-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:38:09,432-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:38:09,531-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:38:19,539-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:38:29,547-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:38:31,458-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:38:39,555-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:38:46,750-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:38:49,563-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:38:59,571-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:39:07,932-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:39:09,580-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:39:19,588-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:39:24,068-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:39:29,596-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:39:39,604-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:39:44,406-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:39:49,612-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:39:59,620-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:40:01,386-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:40:09,628-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:40:19,636-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:40:20,880-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:40:29,644-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:40:38,704-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:40:39,652-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:40:49,660-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:40:57,354-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:40:59,668-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:41:09,676-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:41:16,022-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:41:19,684-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:41:29,693-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:41:33,828-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:41:39,701-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:41:49,709-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:41:53,340-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:41:59,717-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:42:09,725-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 05:42:10,301-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 05:42:19,733-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:42:29,741-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:42:30,659-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:42:39,748-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:42:46,776-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:42:49,756-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:42:59,765-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:43:07,976-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:43:09,773-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:43:19,781-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 05:43:23,251-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 05:43:29,789-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:43:39,797-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 05:43:45,295-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 05:43:49,806-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:43:59,724-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:43:59,814-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:44:09,822-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:44:19,830-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:44:22,612-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:44:29,839-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:44:36,198-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:44:39,847-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:44:49,855-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:44:59,864-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 05:44:59,931-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 05:45:09,872-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:45:12,672-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:45:19,880-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:45:29,889-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:45:37,248-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:45:39,897-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 05:45:49,145-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 05:45:49,905-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:45:59,913-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:46:09,921-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:46:14,566-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:46:19,929-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:46:25,620-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:46:29,938-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:46:39,946-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:46:49,954-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:46:51,884-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:46:59,963-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:47:02,094-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:47:09,971-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:47:19,979-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:47:29,202-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:47:29,987-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:47:38,568-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:47:39,996-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:47:50,004-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:48:00,012-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 05:48:06,521-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 05:48:10,020-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 05:48:15,041-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 05:48:20,028-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:48:30,037-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:48:40,045-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:48:43,838-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:48:50,053-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:48:51,516-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:49:00,062-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:49:10,070-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:49:20,078-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:49:21,156-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:49:27,990-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:49:30,087-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:49:40,095-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:49:50,103-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 05:49:58,475-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 05:50:00,112-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:50:04,464-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:50:10,120-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:50:20,128-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:50:30,137-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:50:35,792-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:50:40,145-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:50:40,938-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:50:50,153-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:51:00,162-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:51:10,169-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:51:13,111-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:51:17,412-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:51:20,177-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:51:30,185-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:51:40,193-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:51:50,201-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 05:51:50,430-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:51:53,886-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:52:00,209-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:52:10,217-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:52:20,225-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:52:27,747-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:52:30,233-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:52:30,360-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:52:40,241-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:52:50,249-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:53:00,257-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 05:53:05,066-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:53:06,834-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:53:10,265-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:53:20,273-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:53:30,281-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:53:40,289-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:53:42,383-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:53:43,308-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:53:50,297-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:54:00,306-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:54:10,314-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 05:54:19,702-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 05:54:19,781-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 05:54:20,322-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:54:30,330-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:54:40,339-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:54:50,347-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:54:56,256-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 05:54:57,020-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 05:55:00,355-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:55:10,363-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:55:20,372-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:55:30,380-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 05:55:32,731-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:55:34,337-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:55:40,388-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:55:50,396-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:56:00,405-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:56:09,204-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:56:10,413-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 05:56:11,656-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 05:56:20,421-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:56:30,430-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:56:40,438-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:56:45,678-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 05:56:48,974-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 05:56:50,447-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:57:00,455-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:57:10,464-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:57:20,472-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:57:22,152-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:57:26,292-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:57:30,480-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:57:40,489-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:57:50,497-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 05:57:58,627-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 05:58:00,505-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-08 05:58:03,611-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-No streams
-2013-11-08 05:58:10,514-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:58:20,523-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:58:30,532-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 05:58:35,100-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 05:58:40,542-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:58:40,928-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 05:58:50,552-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:59:00,563-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:59:10,575-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 05:59:11,574-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 05:59:18,247-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 05:59:20,585-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:59:30,596-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 05:59:40,607-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 05:59:48,049-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 05:59:50,617-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 05:59:55,565-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 06:00:00,626-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:00:10,637-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:00:20,650-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 06:00:24,523-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 06:00:30,662-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 06:00:32,883-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 06:00:40,673-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:00:50,688-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:01:00,706-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 06:01:00,996-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 06:01:10,200-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 06:01:10,719-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:01:20,739-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:01:30,752-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 06:01:37,470-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 06:01:40,767-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-08 06:01:47,520-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-No streams
-2013-11-08 06:01:50,782-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:02:00,791-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:02:10,802-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 06:02:13,945-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 06:02:20,811-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 06:02:24,838-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 06:02:30,821-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:02:40,834-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 06:02:50,418-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 06:02:50,846-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:03:00,857-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 06:03:02,156-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 06:03:10,868-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:03:20,881-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 06:03:26,893-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 06:03:30,892-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 06:03:39,474-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 06:03:40,903-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:03:50,912-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:04:00,921-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:04:03,366-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:04:10,931-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 06:04:16,792-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 06:04:20,940-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:04:30,951-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 06:04:39,841-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 06:04:40,960-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:04:50,968-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 06:04:54,110-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 06:05:00,977-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:05:10,986-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 06:05:16,314-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 06:05:20,996-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:05:31,005-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 06:05:31,429-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 06:05:41,013-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:05:51,022-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 06:05:52,789-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 06:06:01,032-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 06:06:08,746-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 06:06:11,040-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:06:21,050-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:06:29,262-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:06:31,060-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:06:41,069-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-08 06:06:46,065-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-No streams
-2013-11-08 06:06:51,072-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:07:01,079-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 06:07:05,736-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 06:07:11,086-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:07:21,106-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 06:07:23,383-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 06:07:31,119-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:07:41,130-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 06:07:42,211-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 06:07:51,140-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 06:08:00,701-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 06:08:01,150-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:08:11,159-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 06:08:18,684-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 06:08:21,168-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:08:31,176-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 06:08:38,018-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 06:08:41,184-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:08:51,193-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 06:08:55,159-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 06:09:01,204-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:09:11,216-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 06:09:15,337-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 06:09:21,227-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:09:31,236-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 06:09:31,632-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 06:09:41,245-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:09:51,257-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 06:09:52,655-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 06:10:01,268-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 06:10:08,107-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 06:10:11,278-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:10:21,287-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 06:10:29,972-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 06:10:31,297-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:10:41,307-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:10:44,580-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:10:51,315-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:11:01,324-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 06:11:07,291-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 06:11:11,336-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:11:21,054-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:11:21,346-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:11:31,355-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:11:41,364-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 06:11:44,609-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 06:11:51,373-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 06:11:57,529-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 06:12:01,383-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:12:11,391-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:12:21,399-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 06:12:21,927-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 06:12:31,408-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:12:34,002-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:12:41,417-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:12:51,425-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 06:12:59,245-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 06:13:01,433-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:13:10,476-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:13:11,442-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:13:21,452-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:13:31,462-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:13:36,562-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:13:41,473-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-08 06:13:46,951-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-No streams
-2013-11-08 06:13:51,475-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:14:01,484-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:14:11,493-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 06:14:13,880-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 06:14:21,502-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 06:14:23,425-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 06:14:31,510-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:14:41,519-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 06:14:51,198-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 06:14:51,528-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:14:59,898-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:15:01,537-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:15:11,546-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:15:21,554-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:15:28,516-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:15:31,562-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 06:15:36,372-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 06:15:41,571-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:15:51,580-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:16:01,590-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 06:16:05,835-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 06:16:11,598-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 06:16:12,847-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 06:16:21,606-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:16:31,615-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:16:41,624-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 06:16:43,153-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 06:16:49,320-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 06:16:51,633-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:17:01,641-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:17:11,649-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 06:17:20,471-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 06:17:21,657-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:17:25,794-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:17:31,664-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:17:41,672-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:17:51,681-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 06:17:57,790-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 06:18:01,689-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:18:02,268-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:18:11,698-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:18:21,706-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:18:31,716-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:18:35,107-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 06:18:38,743-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 06:18:41,725-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:18:51,734-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:19:01,743-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:19:11,752-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-2013-11-08 06:19:12,427-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:19:15,216-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:19:21,761-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:19:31,769-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:19:41,778-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 06:19:49,744-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 06:19:51,691-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 06:19:51,788-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:20:01,796-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:20:11,806-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:20:21,815-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 06:20:27,061-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 06:20:28,166-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 06:20:31,824-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:20:41,833-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:20:51,841-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:21:01,850-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 06:21:04,380-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:21:04,639-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:21:11,858-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:21:21,867-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:21:31,877-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 06:21:41,114-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:21:41,697-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:21:41,887-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:21:51,895-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:22:01,904-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:22:11,913-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 06:22:17,588-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 06:22:19,015-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 06:22:21,923-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:22:31,934-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:22:41,945-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:22:51,955-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 06:22:54,061-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 06:22:56,334-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 06:23:01,964-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:23:11,974-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:23:21,983-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 06:23:30,536-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 06:23:31,993-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:23:33,651-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:23:42,002-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:23:52,012-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:24:02,020-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 06:24:07,010-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 06:24:10,970-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 06:24:12,030-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:24:22,040-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:24:32,049-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:24:42,059-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 06:24:43,485-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 06:24:48,288-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 06:24:52,069-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:25:02,079-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:25:12,088-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 06:25:19,959-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 06:25:22,097-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 06:25:25,606-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 06:25:32,105-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:25:42,114-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:25:52,123-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 06:25:56,433-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 06:26:02,133-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:26:02,923-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:26:12,141-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:26:22,150-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:26:32,159-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:26:32,906-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 06:26:40,242-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 06:26:42,168-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:26:52,177-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:27:02,186-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 06:27:09,381-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 06:27:12,195-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:27:17,559-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:27:22,204-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:27:32,212-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:27:42,221-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 06:27:45,855-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 06:27:52,229-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 06:27:54,877-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 06:28:02,238-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:28:12,246-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:28:22,254-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 06:28:22,328-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:28:32,195-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:28:32,262-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:28:42,270-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:28:52,278-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 06:28:58,803-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 06:29:02,287-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:29:09,513-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:29:12,295-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:29:22,303-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:29:32,311-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 06:29:35,276-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 06:29:42,320-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 06:29:46,832-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 06:29:52,328-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:30:02,336-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:30:11,750-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:30:12,345-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:30:22,353-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:30:24,149-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:30:32,361-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:30:42,369-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:30:48,224-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:30:52,377-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 06:31:01,468-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 06:31:02,385-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:31:12,393-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:31:22,401-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:31:24,698-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:31:32,409-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:31:38,785-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:31:42,417-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:31:52,425-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 06:32:01,172-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 06:32:02,433-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:32:12,441-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:32:16,104-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:32:22,450-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:32:32,458-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:32:37,646-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:32:42,466-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:32:52,474-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:32:53,422-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:33:02,482-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:33:12,490-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 06:33:14,119-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 06:33:22,498-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:33:30,740-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:33:32,506-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:33:42,514-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:33:50,594-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:33:52,522-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:34:02,530-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:34:08,058-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:34:12,538-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:34:22,546-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:34:27,068-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:34:32,554-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:34:42,562-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:34:45,376-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:34:52,570-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:35:02,578-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-2013-11-08 06:35:03,544-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-No streams
-2013-11-08 06:35:12,587-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:35:22,595-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 06:35:22,695-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 06:35:32,603-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 06:35:40,017-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 06:35:42,611-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:35:52,619-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:36:00,012-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:36:02,627-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:36:12,635-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:36:16,490-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:36:22,643-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:36:32,651-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:36:37,330-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:36:42,660-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:36:52,667-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:36:52,964-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:37:02,676-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:37:12,684-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:37:14,648-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:37:22,692-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:37:29,438-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:37:32,700-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:37:42,708-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:37:51,966-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:37:52,716-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:38:02,724-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:38:05,912-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:38:12,733-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:38:22,741-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:38:29,284-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:38:32,749-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 06:38:42,386-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 06:38:42,758-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:38:52,766-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:39:02,774-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:39:06,602-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:39:12,782-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:39:18,860-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:39:22,790-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:39:32,798-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:39:42,807-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:39:43,920-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:39:52,815-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:39:55,334-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:40:02,823-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:40:12,832-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:40:21,238-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:40:22,840-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:40:31,808-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:40:32,848-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:40:42,856-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:40:52,864-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 06:40:58,556-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 06:41:02,872-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 06:41:08,282-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 06:41:12,880-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:41:22,888-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:41:32,897-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:41:35,874-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:41:42,905-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 06:41:44,757-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 06:41:52,913-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:42:02,921-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:42:12,929-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:42:13,192-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 06:42:21,230-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 06:42:22,938-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:42:32,946-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:42:42,954-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:42:50,510-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:42:52,962-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 06:42:57,704-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 06:43:02,970-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:43:12,978-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:43:22,986-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:43:27,828-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:43:32,994-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 06:43:34,178-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 06:43:43,003-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:43:53,011-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:44:03,019-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:44:05,146-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:44:10,653-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:44:13,028-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:44:23,036-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:44:33,044-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:44:42,464-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:44:43,052-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:44:47,127-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:44:53,060-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:45:03,069-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:45:13,077-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 06:45:19,783-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 06:45:23,085-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:45:23,601-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:45:33,094-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:45:43,102-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:45:53,110-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:45:57,100-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:46:00,075-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:46:03,118-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:46:13,127-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:46:23,135-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:46:33,143-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:46:34,419-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:46:36,549-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:46:43,151-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:46:53,159-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:47:03,167-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:47:11,737-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:47:13,023-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:47:13,176-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:47:23,184-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:47:33,192-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:47:43,200-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 06:47:49,055-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:47:49,497-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:47:53,208-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:48:03,216-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:48:13,224-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:48:23,232-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:48:25,971-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:48:26,373-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:48:33,241-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:48:43,249-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:48:53,257-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 06:49:02,445-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 06:49:03,265-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:49:03,691-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:49:13,274-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:49:23,282-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:49:33,290-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:49:38,920-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 06:49:41,010-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 06:49:43,298-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:49:53,306-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:50:03,314-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:50:13,322-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 06:50:15,395-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:50:18,327-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:50:23,331-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:50:33,339-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:50:43,347-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:50:51,868-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:50:53,355-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:50:55,646-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:51:03,363-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:51:13,372-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:51:23,380-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:51:28,342-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:51:32,964-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:51:33,388-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:51:43,396-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:51:53,405-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:52:03,413-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:52:04,816-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:52:10,282-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:52:13,421-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:52:23,430-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:52:33,438-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:52:41,290-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:52:43,446-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:52:47,600-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:52:53,454-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:53:03,462-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:53:13,470-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 06:53:17,763-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 06:53:23,479-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 06:53:24,919-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 06:53:33,487-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:53:43,495-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:53:53,503-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:53:54,238-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:54:02,236-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:54:03,512-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:54:13,520-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:54:23,528-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 06:54:30,711-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 06:54:33,536-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:54:39,554-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:54:43,544-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:54:53,552-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:55:03,561-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 06:55:07,187-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 06:55:13,569-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 06:55:16,873-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 06:55:23,577-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:55:33,585-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:55:43,594-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 06:55:43,661-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 06:55:53,602-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:55:54,190-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:56:03,610-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:56:13,618-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:56:20,135-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:56:23,627-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:56:31,509-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:56:33,635-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:56:43,643-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:56:53,651-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:56:56,609-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:57:03,659-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:57:08,827-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:57:13,667-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:57:23,676-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 06:57:33,083-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 06:57:33,684-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:57:43,692-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 06:57:46,145-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 06:57:53,700-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:58:03,708-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:58:09,557-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:58:13,716-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:58:23,463-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:58:23,724-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:58:33,732-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:58:43,740-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 06:58:46,031-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 06:58:53,749-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:59:00,781-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:59:03,757-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:59:13,765-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:59:22,505-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:59:23,774-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:59:33,782-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 06:59:38,099-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 06:59:43,790-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 06:59:53,799-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 06:59:58,978-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 07:00:03,807-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:00:13,815-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:00:15,417-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 07:00:23,824-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:00:33,832-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 07:00:35,454-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 07:00:43,840-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:00:52,735-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 07:00:53,848-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:01:03,857-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:01:11,927-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:01:13,865-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:01:23,873-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:01:30,053-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 07:01:33,881-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:01:43,889-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:01:48,401-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:01:53,898-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:02:03,906-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 07:02:07,372-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 07:02:13,914-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:02:23,922-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 07:02:24,875-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 07:02:33,936-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:02:43,944-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:02:44,689-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 07:02:53,952-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:03:01,350-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:03:03,960-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:03:13,968-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 07:03:22,008-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 07:03:23,976-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:03:33,984-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:03:37,824-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 07:03:43,992-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:03:54,000-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:03:59,325-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 07:04:04,008-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:04:14,016-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:04:14,298-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:04:24,024-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:04:34,032-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:04:36,643-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 07:04:44,040-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:04:50,772-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 07:04:54,048-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:05:04,056-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:05:13,961-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 07:05:14,065-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:05:24,073-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:05:27,246-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 07:05:34,081-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:05:44,089-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:05:51,279-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 07:05:54,097-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:06:03,720-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 07:06:04,105-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:06:14,113-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:06:24,121-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:06:28,597-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 07:06:34,129-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:06:40,194-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:06:44,137-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:06:54,145-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:07:04,154-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:07:05,915-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 07:07:14,162-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:07:16,668-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 07:07:24,170-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:07:34,178-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 07:07:43,234-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 07:07:44,186-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 07:07:53,141-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 07:07:54,194-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:08:04,202-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:08:14,210-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:08:20,551-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 07:08:24,218-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:08:29,616-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 07:08:34,226-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:08:44,234-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:08:54,242-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 07:08:57,870-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 07:09:04,250-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:09:06,091-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:09:14,258-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:09:24,266-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:09:34,274-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:09:35,187-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:09:42,564-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 07:09:44,282-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:09:54,290-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:10:04,298-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:10:12,505-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 07:10:14,306-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:10:19,038-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 07:10:24,314-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:10:34,322-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:10:44,330-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 07:10:49,824-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 07:10:54,339-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 07:10:55,513-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 07:11:04,347-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:11:14,355-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:11:24,363-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:11:27,141-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 07:11:31,986-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 07:11:34,371-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:11:44,379-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:11:54,387-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:12:04,395-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 07:12:04,460-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:12:08,461-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:12:14,404-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:12:24,412-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:12:34,420-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:12:41,777-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 07:12:44,428-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:12:44,935-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 07:12:54,436-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:13:04,445-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:13:14,452-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 07:13:19,095-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:13:21,409-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 07:13:24,460-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:13:34,468-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:13:44,477-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:13:54,485-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:13:56,414-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:13:57,883-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 07:14:04,493-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:14:14,501-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:14:24,509-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:14:33,732-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:14:34,357-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:14:34,518-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:14:44,526-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:14:54,534-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:15:04,542-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:15:10,831-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:15:11,050-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 07:15:14,550-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:15:24,558-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:15:34,566-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:15:44,575-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:15:47,305-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:15:48,368-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 07:15:54,583-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:16:04,591-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:16:14,599-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:16:23,779-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:16:24,608-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:16:25,686-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 07:16:34,616-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:16:44,624-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:16:54,632-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:17:00,253-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:17:03,004-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 07:17:04,641-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:17:14,649-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:17:24,657-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:17:34,665-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 07:17:36,728-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:17:40,322-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 07:17:44,673-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:17:54,681-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:18:04,689-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:18:13,201-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:18:14,697-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 07:18:17,641-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 07:18:24,704-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:18:34,714-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:18:44,722-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:18:49,675-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:18:54,729-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:18:54,958-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 07:19:04,737-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:19:14,745-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:19:24,753-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:19:26,149-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:19:32,277-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:19:34,762-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:19:44,769-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:19:54,778-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:20:02,623-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:20:04,786-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:20:09,595-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 07:20:14,794-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:20:24,802-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:20:34,810-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:20:39,098-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:20:44,818-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:20:46,913-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:20:54,826-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:21:04,834-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:21:14,842-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:21:15,572-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:21:24,231-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 07:21:24,850-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:21:34,860-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:21:44,869-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 07:21:52,047-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 07:21:54,877-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:22:01,549-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 07:22:04,885-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:22:14,893-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:22:24,901-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:22:28,520-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:22:34,909-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:22:38,867-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 07:22:44,917-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:22:54,925-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:23:04,934-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 07:23:04,993-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 07:23:14,942-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:23:16,185-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 07:23:24,950-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:23:34,958-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:23:41,468-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:23:44,966-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:23:53,503-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:23:54,974-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:24:04,982-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:24:14,990-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 07:24:17,943-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 07:24:24,998-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:24:30,821-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 07:24:35,006-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:24:45,014-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:24:54,416-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:24:55,022-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:25:05,030-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:25:08,139-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:25:15,040-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:25:25,048-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 07:25:30,891-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 07:25:35,056-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:25:45,064-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:25:45,457-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 07:25:55,072-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:26:05,080-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:26:07,364-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:26:15,089-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:26:22,775-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:26:25,097-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:26:35,105-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:26:43,838-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 07:26:45,113-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:26:55,121-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 07:27:00,094-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 07:27:05,129-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:27:15,138-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:27:20,312-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:27:25,145-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:27:35,154-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:27:37,411-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 07:27:45,162-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:27:55,170-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:27:56,786-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 07:28:05,178-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:28:14,730-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:28:15,187-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:28:25,195-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:28:33,260-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:28:35,203-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:28:45,211-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:28:52,048-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 07:28:55,220-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:29:05,228-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:29:09,734-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:29:15,236-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:29:25,244-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:29:29,366-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:29:35,252-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:29:45,260-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:29:46,208-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:29:55,268-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:30:05,276-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 07:30:06,685-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 07:30:15,284-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 07:30:22,683-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 07:30:25,293-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:30:35,301-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:30:44,002-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:30:45,309-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:30:55,317-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:30:59,156-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:31:05,325-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:31:15,334-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:31:21,320-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:31:25,342-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:31:35,350-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:31:35,630-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 07:31:45,358-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:31:55,366-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:31:58,638-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:32:05,375-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:32:12,104-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:32:15,383-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:32:25,391-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:32:35,399-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 07:32:35,957-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 07:32:45,407-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:32:48,578-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:32:55,415-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:33:05,424-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:33:13,274-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 07:33:15,432-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 07:33:25,051-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 07:33:25,440-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:33:35,448-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:33:45,456-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:33:50,592-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:33:55,463-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:34:01,526-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:34:05,471-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:34:15,479-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:34:25,488-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:34:27,910-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 07:34:35,496-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 07:34:37,999-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 07:34:45,504-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:34:55,512-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:35:05,228-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 07:35:05,520-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:35:14,474-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:35:15,528-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:35:25,535-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:35:35,543-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:35:42,546-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:35:45,551-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:35:50,948-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:35:55,559-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:36:05,567-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:36:15,575-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 07:36:19,865-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 07:36:25,583-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 07:36:27,423-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 07:36:35,591-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:36:45,600-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:36:55,608-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:36:57,182-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:37:03,896-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:37:05,616-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:37:15,624-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:37:25,632-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:37:34,500-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:37:35,640-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 07:37:40,369-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 07:37:45,648-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:37:55,656-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:38:05,664-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 07:38:11,819-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 07:38:15,672-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:38:16,844-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:38:25,680-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:38:35,689-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:38:45,697-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:38:49,136-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 07:38:53,317-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 07:38:55,704-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:39:05,713-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:39:15,721-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:39:25,729-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:39:26,454-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:39:29,792-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:39:35,737-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:39:45,745-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:39:55,753-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:40:03,772-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:40:05,761-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:40:06,266-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 07:40:15,768-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:40:25,776-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:40:35,784-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:40:41,090-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:40:42,740-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:40:45,792-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:40:55,800-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:41:05,809-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:41:15,816-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 07:41:18,409-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:41:19,214-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:41:25,824-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:41:35,832-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:41:45,840-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:41:55,688-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:41:55,726-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:41:55,848-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:42:05,855-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:42:15,863-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:42:25,871-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:42:32,162-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:42:33,044-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:42:35,879-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:42:45,887-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:42:55,895-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:43:05,902-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:43:08,636-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:43:10,362-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:43:15,910-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:43:25,918-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:43:35,926-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:43:45,110-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:43:45,934-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 07:43:47,681-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 07:43:55,941-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:44:05,949-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:44:15,957-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:44:21,584-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:44:24,998-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:44:25,966-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:44:35,973-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:44:45,981-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:44:55,989-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:44:58,058-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 07:45:02,317-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 07:45:05,996-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:45:16,005-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:45:26,013-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 07:45:34,533-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 07:45:36,022-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:45:39,634-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 07:45:46,030-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:45:56,038-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:46:06,045-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:46:11,006-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:46:16,054-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:46:16,952-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:46:26,064-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:46:36,072-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:46:46,080-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 07:46:47,481-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:46:54,270-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 07:46:56,087-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:47:06,095-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:47:16,103-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:47:23,954-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:47:26,112-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:47:31,588-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:47:36,120-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:47:46,128-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:47:56,136-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:48:00,428-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:48:06,144-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:48:08,906-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:48:16,152-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:48:26,160-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:48:36,169-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 07:48:36,901-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 07:48:46,177-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:48:46,224-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:48:56,185-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:49:06,193-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:49:13,376-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:49:16,202-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:49:23,542-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 07:49:26,210-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:49:36,218-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:49:46,226-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:49:49,850-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 07:49:56,234-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:50:00,860-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:50:06,242-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:50:16,250-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:50:26,259-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:50:26,324-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:50:36,267-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 07:50:38,179-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 07:50:46,276-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:50:56,284-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:51:02,798-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 07:51:06,292-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:51:15,496-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:51:16,299-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:51:26,307-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:51:36,315-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:51:39,272-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:51:46,323-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 07:51:52,815-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 07:51:56,331-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:52:06,339-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:52:15,746-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:52:16,346-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:52:26,354-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:52:30,132-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:52:36,362-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:52:46,370-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:52:52,220-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:52:56,378-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:53:06,386-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:53:07,451-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:53:16,394-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:53:26,402-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 07:53:28,695-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 07:53:36,410-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:53:44,769-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 07:53:46,418-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:53:56,426-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:54:05,168-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:54:06,434-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:54:16,442-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:54:22,088-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:54:26,452-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:54:36,460-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:54:41,642-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:54:46,468-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:54:56,476-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:54:59,405-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 07:55:06,485-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:55:16,493-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 07:55:18,117-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 07:55:26,501-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:55:36,509-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:55:36,724-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:55:46,518-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:55:54,590-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:55:56,526-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:56:06,534-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:56:14,042-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:56:16,543-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:56:26,551-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:56:31,064-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:56:36,559-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:56:46,567-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:56:51,360-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:56:56,575-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:57:06,583-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 07:57:07,538-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 07:57:16,591-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:57:26,599-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:57:28,678-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:57:36,606-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:57:44,012-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:57:46,614-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:57:56,622-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:58:05,996-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:58:06,630-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:58:16,638-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:58:20,486-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:58:26,646-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:58:36,654-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:58:43,314-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 07:58:46,662-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:58:56,670-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:58:56,960-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:59:06,678-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:59:16,685-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 07:59:20,632-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 07:59:26,694-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:59:33,434-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 07:59:36,702-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:59:46,709-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 07:59:56,717-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 07:59:57,950-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:00:06,725-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 87ms
-2013-11-08 08:00:09,994-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 87ms
-No streams
-2013-11-08 08:00:16,733-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:00:26,741-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:00:35,268-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 08:00:36,749-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:00:46,382-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 08:00:46,756-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:00:56,764-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:01:06,773-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:01:12,586-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:01:16,780-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 08:01:22,856-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 08:01:26,788-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:01:36,796-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:01:46,804-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:01:49,904-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 08:01:56,811-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:01:59,331-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 08:02:06,819-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:02:16,827-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:02:26,835-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 08:02:27,223-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:02:35,805-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 08:02:36,843-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:02:46,851-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:02:56,859-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:03:04,540-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 08:03:06,866-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:03:12,279-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 08:03:16,874-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:03:26,882-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:03:36,892-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:03:41,858-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:03:46,899-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:03:48,753-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 08:03:56,907-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:04:06,915-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:04:16,922-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:04:19,176-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 08:04:25,228-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 08:04:26,931-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:04:36,938-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:04:46,946-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:04:56,494-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:04:56,954-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:05:01,702-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 08:05:06,962-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:05:16,970-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:05:26,977-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:05:33,812-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 08:05:36,985-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:05:38,176-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:05:46,993-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:05:57,001-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:06:07,009-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 08:06:11,131-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:06:14,650-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 08:06:17,017-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:06:27,025-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:06:37,033-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:06:47,040-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:06:48,448-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:06:51,124-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 08:06:57,049-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:07:07,057-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:07:17,064-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:07:25,766-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:07:27,072-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 08:07:27,598-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 08:07:37,080-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:07:47,088-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:07:57,096-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:08:03,084-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:08:04,073-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 08:08:07,104-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:08:17,112-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:08:27,120-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:08:37,127-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 08:08:40,403-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:08:40,547-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:08:47,135-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:08:57,143-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:09:07,151-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:09:17,021-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 08:09:17,159-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:09:17,720-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:09:27,167-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:09:37,175-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:09:47,183-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:09:53,495-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 08:09:55,039-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 08:09:57,191-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:10:07,199-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:10:17,207-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:10:27,215-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:10:29,969-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:10:32,356-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:10:37,223-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:10:47,231-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:10:57,239-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:11:06,443-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 08:11:07,247-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:11:09,674-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 08:11:17,255-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:11:27,263-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:11:37,270-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 08:11:42,917-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-2013-11-08 08:11:46,994-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-No streams
-2013-11-08 08:11:47,278-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:11:57,286-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:12:07,294-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:12:17,302-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:12:19,392-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:12:24,310-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 08:12:27,310-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:12:37,317-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:12:47,326-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:12:55,866-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 08:12:57,334-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:13:01,628-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:13:07,341-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:13:17,349-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:13:27,357-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:13:32,340-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 08:13:37,367-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:13:38,946-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 08:13:47,374-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:13:57,381-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:14:07,389-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:14:08,814-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:14:16,264-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:14:17,397-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:14:27,405-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:14:37,413-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:14:45,288-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 08:14:47,421-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:14:53,582-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 08:14:57,428-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:15:07,436-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:15:17,445-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:15:21,762-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 08:15:27,453-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 08:15:30,901-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 08:15:37,461-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:15:47,468-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:15:57,476-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 08:15:58,237-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 08:16:07,485-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:16:08,218-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:16:17,493-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:16:27,500-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:16:34,710-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 08:16:37,509-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 08:16:45,537-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 08:16:47,516-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:16:57,524-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:17:07,532-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:17:11,184-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 08:17:17,540-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:17:22,854-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:17:27,548-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:17:37,556-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:17:47,564-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:17:47,658-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 08:17:57,572-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:18:00,172-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 08:18:07,580-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:18:17,588-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:18:24,133-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 08:18:27,596-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:18:37,490-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:18:37,604-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:18:47,612-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:18:57,620-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:19:00,607-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 08:19:07,628-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:19:14,808-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 08:19:17,636-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:19:27,644-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:19:37,081-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:19:37,652-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:19:47,660-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 08:19:52,127-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 08:19:57,668-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:20:07,676-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:20:13,555-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 08:20:17,685-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:20:27,692-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:20:29,444-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:20:37,700-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:20:47,708-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:20:50,029-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 08:20:57,716-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 08:21:06,763-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 08:21:07,725-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:21:17,733-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:21:26,503-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 08:21:27,741-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:21:37,748-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:21:44,080-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 08:21:47,756-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:21:57,764-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:22:02,977-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:22:07,772-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:22:17,780-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 08:22:21,399-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 08:22:27,788-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:22:37,796-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 08:22:39,450-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 08:22:47,805-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:22:57,813-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:22:58,716-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 08:23:07,821-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:23:15,925-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:23:17,829-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:23:27,836-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:23:36,035-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 08:23:37,845-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:23:47,853-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 08:23:52,399-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 08:23:57,861-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:24:07,868-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:24:13,353-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:24:17,876-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:24:27,884-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:24:28,874-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 08:24:37,892-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:24:47,900-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 08:24:50,672-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 08:24:57,908-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 08:25:05,349-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 08:25:07,916-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:25:17,924-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:25:27,932-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 08:25:27,990-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 08:25:37,940-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:25:41,822-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 08:25:47,949-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:25:57,957-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:26:05,307-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:26:07,966-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:26:17,974-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:26:18,296-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:26:27,982-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:26:37,991-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:26:42,625-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:26:47,999-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:26:54,770-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 08:26:58,007-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:27:08,015-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:27:18,024-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:27:19,943-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:27:28,032-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:27:31,244-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 08:27:38,040-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:27:48,049-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 08:27:57,262-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 08:27:58,057-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:28:07,718-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 08:28:08,065-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:28:18,073-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:28:28,081-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:28:34,579-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:28:38,089-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:28:44,192-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:28:48,097-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:28:58,105-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:29:08,114-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 08:29:11,898-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 08:29:18,121-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 08:29:20,665-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 08:29:28,129-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:29:38,137-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:29:48,146-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:29:49,215-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:29:57,140-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:29:58,154-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:30:08,162-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:30:18,171-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:30:26,534-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 08:30:28,179-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 08:30:33,615-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 08:30:38,187-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:30:48,196-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:30:58,204-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:31:03,852-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:31:08,212-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:31:10,088-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 08:31:18,221-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:31:28,229-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:31:38,238-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:31:41,170-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:31:46,562-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:31:48,246-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:31:58,254-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:32:08,263-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:32:18,271-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:32:18,488-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:32:23,036-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 08:32:28,279-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:32:38,288-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:32:48,296-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:32:55,806-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:32:58,304-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:32:59,510-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:33:08,313-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:33:18,321-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:33:28,329-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:33:33,124-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 08:33:35,983-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 08:33:38,338-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:33:48,347-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:33:58,355-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:34:08,363-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:34:10,442-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:34:12,458-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:34:18,372-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:34:28,381-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:34:38,389-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 08:34:47,761-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 08:34:48,398-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:34:48,932-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 08:34:58,406-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:35:08,414-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:35:18,422-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 08:35:25,079-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:35:25,406-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:35:28,430-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:35:38,438-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:35:48,445-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:35:58,453-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:36:01,880-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:36:02,396-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:36:08,462-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:36:18,470-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:36:28,477-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:36:38,354-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:36:38,485-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 08:36:39,715-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 08:36:48,493-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:36:58,502-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:37:08,510-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:37:14,828-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:37:17,032-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:37:18,518-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:37:28,525-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:37:38,533-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:37:48,541-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:37:51,302-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:37:54,350-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:37:58,550-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:38:08,558-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:38:18,566-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:38:27,777-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:38:28,574-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:38:31,668-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:38:38,582-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:38:48,591-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:38:58,599-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:39:04,250-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:39:08,607-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:39:08,986-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 08:39:18,615-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:39:28,625-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:39:38,633-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:39:40,725-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:39:46,304-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:39:48,642-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:39:58,650-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:40:08,659-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 08:40:17,200-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 08:40:18,667-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:40:23,622-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:40:28,676-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:40:38,684-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:40:48,693-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:40:53,673-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 08:40:58,701-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:41:00,940-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:41:08,709-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:41:18,717-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:41:28,726-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:41:30,147-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:41:38,258-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:41:38,735-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:41:48,743-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:41:58,751-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:42:06,621-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:42:08,761-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:42:15,576-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:42:18,769-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:42:28,777-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:42:38,785-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 08:42:43,096-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 08:42:48,793-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:42:52,895-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:42:58,802-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:43:08,810-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:43:18,818-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 08:43:19,568-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 08:43:28,827-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:43:30,213-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:43:38,835-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:43:48,843-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:43:56,043-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:43:58,851-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:44:07,531-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:44:08,859-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:44:18,868-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:44:28,876-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:44:32,517-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 08:44:38,884-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 08:44:44,850-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 08:44:48,892-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:44:58,900-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:45:08,909-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:45:08,991-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:45:18,917-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:45:22,167-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:45:28,925-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:45:38,933-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:45:45,465-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 08:45:48,940-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:45:58,949-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:45:59,485-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:46:08,957-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:46:18,965-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 08:46:21,940-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 08:46:28,973-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:46:36,803-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:46:38,981-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:46:48,989-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:46:58,413-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 08:46:58,997-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:47:09,005-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:47:14,121-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:47:19,013-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:47:29,021-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:47:34,887-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:47:39,029-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:47:49,037-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:47:51,439-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:47:59,045-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:48:09,054-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 08:48:11,360-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 08:48:19,062-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:48:28,757-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:48:29,070-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:48:39,078-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:48:47,835-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:48:49,086-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:48:59,094-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 08:49:06,076-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 08:49:09,103-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:49:19,111-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:49:24,309-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 08:49:29,120-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:49:39,128-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:49:43,393-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:49:49,135-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:49:59,143-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:50:00,783-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 08:50:09,151-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:50:19,159-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 08:50:20,712-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 08:50:29,167-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:50:37,257-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 08:50:39,175-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:50:49,183-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:50:58,029-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:50:59,191-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:51:09,199-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:51:13,731-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:51:19,207-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:51:29,214-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:51:35,347-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:51:39,222-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:51:49,230-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 08:51:50,206-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 08:51:59,239-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:52:09,248-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:52:12,665-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:52:19,255-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:52:26,679-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 08:52:29,263-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:52:39,271-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:52:49,280-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:52:49,983-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:52:59,287-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:53:03,154-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 08:53:09,296-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:53:19,304-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:53:27,302-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:53:29,311-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:53:39,319-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:53:39,628-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:53:49,327-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:53:59,335-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:54:04,619-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:54:09,344-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 08:54:16,102-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 08:54:19,352-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:54:29,360-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:54:39,368-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:54:41,937-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:54:49,376-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:54:52,577-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:54:59,384-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:55:09,392-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:55:19,255-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:55:19,399-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 08:55:29,052-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 08:55:29,407-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:55:39,415-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:55:49,422-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:55:56,573-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:55:59,430-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:56:05,525-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 08:56:09,438-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:56:19,445-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:56:29,453-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:56:33,891-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:56:39,461-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:56:41,999-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:56:49,468-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:56:59,477-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:57:09,485-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:57:11,209-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:57:18,473-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 08:57:19,493-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:57:29,501-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:57:39,510-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 08:57:48,528-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 08:57:49,518-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:57:54,947-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:57:59,526-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:58:09,534-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:58:19,542-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:58:25,845-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:58:29,550-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 08:58:31,420-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 08:58:39,560-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:58:49,568-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:58:59,576-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 08:59:03,164-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:59:07,895-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 08:59:09,584-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:59:19,593-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:59:29,601-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:59:39,609-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 08:59:40,481-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 08:59:44,370-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 08:59:49,618-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 08:59:59,627-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:00:09,635-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 09:00:17,800-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 09:00:19,643-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 09:00:20,844-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 09:00:29,651-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:00:39,659-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:00:49,667-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:00:55,117-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:00:57,319-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:00:59,675-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:01:09,683-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:01:19,692-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:01:29,700-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:01:32,435-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:01:33,793-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:01:39,708-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:01:49,716-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:01:59,724-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:02:09,733-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:02:09,753-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:02:10,267-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:02:19,741-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:02:29,749-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:02:39,757-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 09:02:46,741-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:02:47,071-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:02:49,767-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:02:59,774-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:03:09,783-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:03:19,790-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:03:23,215-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 09:03:24,390-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 09:03:29,798-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:03:39,808-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:03:49,816-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 09:03:59,689-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 09:03:59,824-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:04:01,707-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:04:09,832-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:04:19,841-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:04:29,849-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 09:04:36,163-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 09:04:39,025-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 09:04:39,857-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:04:49,865-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:04:59,874-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:05:09,882-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 09:05:12,638-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:05:16,344-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:05:19,890-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:05:29,898-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:05:39,906-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:05:49,112-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:05:49,915-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:05:53,662-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:05:59,923-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:06:09,931-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:06:19,939-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 09:06:25,586-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 09:06:29,947-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 09:06:30,981-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 09:06:39,955-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:06:49,963-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:06:59,971-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:07:02,061-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 09:07:08,299-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 09:07:09,978-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:07:19,986-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:07:29,994-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-2013-11-08 09:07:38,537-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-No streams
-2013-11-08 09:07:40,001-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 09:07:45,617-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 09:07:50,009-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:08:00,017-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:08:10,025-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 09:08:15,009-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 09:08:20,033-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:08:22,934-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:08:30,041-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:08:40,049-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:08:50,057-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:08:51,483-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:09:00,065-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:09:00,253-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:09:10,073-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:09:20,081-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:09:27,957-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:09:30,090-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:09:37,571-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:09:40,098-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:09:50,106-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:10:00,114-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:10:04,431-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:10:10,122-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:10:14,889-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:10:20,130-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:10:30,138-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:10:40,147-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 09:10:40,905-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 09:10:50,155-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 09:10:52,208-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 09:11:00,163-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:11:10,171-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:11:17,379-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:11:20,179-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:11:29,525-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:11:30,187-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:11:40,195-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:11:50,203-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 09:11:53,853-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 09:12:00,211-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 09:12:06,844-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 09:12:10,220-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:12:20,228-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:12:30,236-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:12:30,327-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:12:40,244-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:12:44,161-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:12:50,253-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:13:00,260-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 09:13:06,801-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 09:13:10,269-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:13:20,277-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:13:21,479-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:13:30,285-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:13:40,293-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:13:43,275-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:13:50,301-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:13:58,797-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:14:00,309-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:14:10,317-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 09:14:19,749-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 09:14:20,325-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:14:30,333-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:14:36,115-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:14:40,342-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:14:50,349-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:14:56,223-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:15:00,358-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:15:10,366-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 09:15:13,434-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 09:15:20,374-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:15:30,382-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:15:32,697-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:15:40,390-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:15:50,398-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:15:50,751-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:16:00,406-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:16:09,171-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:16:10,415-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:16:20,423-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:16:28,069-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:16:30,431-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:16:40,439-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 09:16:45,645-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 09:16:50,447-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:17:00,456-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 09:17:05,388-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 09:17:10,464-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:17:20,472-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 09:17:22,120-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 09:17:30,480-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:17:40,488-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:17:42,705-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:17:50,496-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 09:17:58,593-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 09:18:00,504-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:18:10,512-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:18:20,023-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:18:20,520-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:18:30,528-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:18:35,067-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:18:40,536-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:18:50,544-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:18:57,341-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:19:00,553-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:19:10,560-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 09:19:11,541-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 09:19:20,568-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:19:30,576-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 09:19:34,660-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 09:19:40,584-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:19:48,015-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:19:50,592-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:20:00,599-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:20:10,608-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:20:11,977-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:20:20,616-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 09:20:24,490-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 09:20:30,624-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:20:40,632-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:20:49,295-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:20:50,640-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:21:00,648-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 09:21:00,964-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 09:21:10,656-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:21:20,664-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:21:26,613-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:21:30,672-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 09:21:37,439-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 09:21:40,680-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:21:50,688-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:22:00,696-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 09:22:03,931-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 09:22:10,704-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 09:22:13,912-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 09:22:20,712-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:22:30,720-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:22:40,728-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:22:41,249-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:22:50,386-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:22:50,736-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:23:00,744-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:23:10,752-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:23:18,567-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:23:20,760-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 09:23:26,860-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 09:23:30,768-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:23:40,776-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:23:50,785-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:23:55,885-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:24:00,793-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:24:03,334-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:24:10,801-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:24:20,809-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:24:30,817-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:24:33,203-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 09:24:39,808-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 09:24:40,825-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:24:50,833-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:25:00,842-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 09:25:10,522-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 09:25:10,850-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:25:16,282-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:25:20,858-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:25:30,866-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:25:40,875-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:25:47,839-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:25:50,883-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 09:25:52,756-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 09:26:00,891-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:26:10,899-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:26:20,907-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:26:25,157-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:26:29,230-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:26:30,915-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:26:40,924-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:26:50,932-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:27:00,940-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:27:02,475-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 09:27:05,704-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 09:27:10,948-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:27:20,956-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:27:30,965-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 09:27:39,793-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 09:27:40,973-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:27:42,179-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:27:50,981-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:28:00,990-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:28:10,998-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 09:28:17,112-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 09:28:18,654-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 09:28:21,006-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:28:31,014-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:28:41,022-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:28:51,031-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:28:54,429-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 251ms
-2013-11-08 09:28:55,377-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 251ms
-No streams
-2013-11-08 09:29:01,039-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:29:11,047-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:29:21,055-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:29:31,063-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:29:31,601-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 09:29:31,748-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 09:29:41,072-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:29:51,080-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:30:01,088-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 09:30:08,075-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:30:09,065-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:30:11,096-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:30:21,104-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:30:31,112-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:30:41,120-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 09:30:44,550-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:30:46,383-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:30:51,128-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:31:01,137-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:31:11,145-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 09:31:21,023-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 09:31:21,153-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:31:23,701-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:31:31,161-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:31:41,170-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:31:51,178-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:31:57,497-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:32:01,019-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:32:01,186-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:32:11,194-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:32:21,203-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:32:31,211-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 09:32:33,971-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:32:38,337-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:32:41,219-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:32:51,227-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:33:01,235-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:33:10,445-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:33:11,243-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:33:15,655-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:33:21,252-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:33:31,260-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:33:41,268-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 194ms
-2013-11-08 09:33:47,112-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 194ms
-No streams
-2013-11-08 09:33:51,276-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 09:33:52,974-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 09:34:01,285-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:34:11,293-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:34:21,301-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:34:23,393-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:34:30,291-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:34:31,309-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:34:41,317-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:34:51,325-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 09:34:59,868-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 09:35:01,333-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 09:35:07,610-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 09:35:11,341-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:35:21,349-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:35:31,358-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:35:36,342-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:35:41,366-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:35:44,927-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:35:51,375-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:36:01,383-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:36:11,391-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 09:36:12,816-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 09:36:21,399-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:36:22,245-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:36:31,408-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:36:41,416-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:36:49,290-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:36:51,424-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 09:36:59,564-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 09:37:01,432-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:37:11,440-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:37:21,448-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 09:37:25,763-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 09:37:31,457-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:37:36,882-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:37:41,465-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:37:51,473-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:38:01,481-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:38:02,238-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:38:11,489-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 09:38:14,200-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 09:38:21,497-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:38:31,505-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 09:38:38,712-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 09:38:41,513-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:38:51,518-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:38:51,521-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:39:01,529-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:39:11,537-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:39:15,186-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:39:21,544-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 09:39:28,836-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 09:39:31,552-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:39:41,560-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:39:51,568-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 09:39:51,661-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 09:40:01,576-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 09:40:06,155-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 09:40:11,584-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:40:21,592-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:40:28,135-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:40:31,600-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:40:41,607-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:40:43,472-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:40:51,615-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:41:01,623-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 09:41:04,609-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 09:41:11,630-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 09:41:20,790-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 09:41:21,638-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:41:31,645-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 09:41:41,083-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 09:41:41,654-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:41:51,662-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:41:58,108-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:42:01,670-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:42:11,679-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 09:42:17,557-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 09:42:21,687-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:42:31,695-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 09:42:35,426-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 09:42:41,703-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:42:51,711-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:42:54,032-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:43:01,720-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:43:11,728-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:43:12,744-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:43:21,736-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:43:30,506-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:43:31,744-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:43:41,752-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:43:50,062-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:43:51,760-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:44:01,768-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:44:06,980-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:44:11,776-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:44:21,784-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:44:27,380-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:44:31,792-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:44:41,800-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 09:44:43,455-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 09:44:51,808-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:45:01,817-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:45:04,698-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:45:11,825-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 09:45:19,928-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 09:45:21,833-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:45:31,841-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:45:41,849-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:45:42,016-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:45:51,857-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 09:45:56,402-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 09:46:01,866-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:46:11,874-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 09:46:19,335-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 09:46:21,883-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:46:31,891-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:46:32,876-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:46:41,899-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:46:51,907-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:46:56,652-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:47:01,915-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 09:47:09,350-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 09:47:11,923-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:47:21,931-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:47:31,940-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 09:47:33,971-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 09:47:41,948-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 09:47:45,824-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 09:47:51,956-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:48:01,965-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:48:11,289-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:48:11,973-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:48:21,981-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 09:48:22,299-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 09:48:31,989-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:48:41,998-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 09:48:48,607-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 09:48:52,006-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:48:58,773-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:49:02,014-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:49:12,022-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:49:22,031-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:49:25,925-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:49:32,039-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 09:49:35,248-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 09:49:42,047-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:49:52,055-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:50:02,064-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:50:03,243-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 09:50:11,721-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 09:50:12,072-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:50:22,080-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:50:32,089-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 09:50:40,562-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 09:50:42,097-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 09:50:48,196-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 09:50:52,105-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:51:02,113-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:51:12,121-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:51:17,879-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:51:22,129-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:51:24,669-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:51:32,137-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:51:42,146-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:51:52,154-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 09:51:55,198-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:52:01,143-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:52:02,162-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:52:12,170-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:52:22,179-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:52:32,187-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:52:32,516-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:52:37,617-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:52:42,195-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:52:52,203-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:53:02,211-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:53:09,835-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:53:12,218-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 09:53:14,092-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 09:53:22,226-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:53:32,234-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:53:42,242-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:53:47,153-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 09:53:50,564-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 09:53:52,250-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:54:02,258-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:54:12,265-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:54:22,273-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:54:24,471-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 09:54:27,039-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 09:54:32,281-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:54:42,288-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:54:52,296-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 09:55:01,790-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 09:55:02,304-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 09:55:03,513-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 09:55:12,312-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:55:22,319-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:55:32,327-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:55:39,107-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:55:39,987-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:55:42,335-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:55:52,343-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:56:02,351-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:56:12,358-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:56:16,425-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 09:56:16,460-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 09:56:22,366-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:56:32,374-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:56:42,382-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:56:52,390-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:56:52,935-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:56:53,743-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:57:02,398-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:57:12,406-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:57:22,414-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 09:57:29,409-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:57:31,061-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:57:32,422-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:57:42,430-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:57:52,440-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:58:02,448-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 09:58:05,884-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:58:08,379-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:58:12,456-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:58:22,465-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:58:32,473-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:58:42,357-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:58:42,482-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 09:58:45,698-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 09:58:52,490-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:59:02,498-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:59:12,506-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-2013-11-08 09:59:18,833-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-No streams
-2013-11-08 09:59:22,514-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 09:59:23,015-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 09:59:32,522-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:59:42,530-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 09:59:52,538-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 09:59:55,306-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:00:00,333-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:00:02,546-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:00:12,555-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:00:22,563-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 10:00:31,779-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 10:00:32,571-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 10:00:37,651-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 10:00:42,579-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:00:52,587-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:01:02,595-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 10:01:08,253-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 10:01:12,604-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 10:01:14,969-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 10:01:22,612-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:01:32,620-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:01:42,629-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 10:01:44,727-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:01:52,288-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:01:52,637-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:02:02,646-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:02:12,654-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:02:21,201-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:02:22,663-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 10:02:29,605-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 10:02:32,671-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:02:42,679-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:02:52,687-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 4ms
-2013-11-08 10:02:57,679-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 4ms
-No streams
-2013-11-08 10:03:02,695-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:03:06,923-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:03:12,703-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:03:22,711-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:03:32,720-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 10:03:34,150-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 10:03:42,729-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 10:03:44,241-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 10:03:52,737-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:04:02,746-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:04:10,625-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:04:12,754-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:04:21,559-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:04:22,762-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:04:32,770-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:04:42,779-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:04:47,099-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:04:52,787-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 10:04:58,876-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 10:05:02,795-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:05:12,803-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:05:22,812-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:05:23,574-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:05:32,820-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 10:05:36,196-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 10:05:42,828-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:05:52,837-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 10:06:00,049-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 10:06:02,845-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:06:12,853-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 10:06:13,513-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 10:06:22,861-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:06:32,869-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 10:06:36,521-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 10:06:42,878-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 10:06:50,832-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 10:06:52,886-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:07:02,894-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:07:12,902-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 10:07:12,996-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 10:07:22,910-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 10:07:28,149-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 10:07:32,918-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:07:42,926-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 10:07:49,469-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 10:07:52,934-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:08:02,942-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:08:05,467-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:08:12,950-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:08:22,958-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:08:25,944-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:08:32,967-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 10:08:42,785-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 10:08:42,975-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:08:52,983-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:09:02,418-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:09:02,990-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:09:12,998-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:09:20,103-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:09:23,006-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:09:33,014-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 10:09:38,892-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 10:09:43,022-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:09:53,030-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 10:09:57,422-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 10:10:03,038-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:10:13,046-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:10:15,366-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:10:23,054-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:10:33,063-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 10:10:34,739-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 10:10:43,072-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:10:51,840-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:10:53,080-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:11:03,088-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:11:12,057-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:11:13,097-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:11:23,105-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:11:28,314-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:11:33,113-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:11:43,121-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 10:11:49,375-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 10:11:53,129-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:12:03,137-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:12:04,788-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:12:13,146-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:12:23,154-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:12:26,693-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:12:33,162-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 6ms
-2013-11-08 10:12:41,267-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 6ms
-No streams
-2013-11-08 10:12:43,171-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:12:53,179-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:13:03,187-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 10:13:04,011-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 10:13:13,196-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:13:17,736-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:13:23,204-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:13:33,212-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:13:41,329-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:13:43,220-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:13:53,228-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:13:54,210-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:14:03,236-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:14:13,244-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 10:14:18,647-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 10:14:23,252-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:14:30,684-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:14:33,260-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:14:43,269-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:14:53,277-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 10:14:55,966-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 10:15:03,285-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:15:07,158-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:15:13,293-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:15:23,301-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:15:33,284-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:15:33,309-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:15:43,317-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 10:15:43,632-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 10:15:53,326-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:16:03,334-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 10:16:10,603-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 10:16:13,342-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:16:20,106-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:16:23,350-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:16:33,358-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:16:43,366-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 10:16:47,921-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 10:16:53,374-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:16:56,580-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:17:03,382-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:17:13,390-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:17:23,398-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:17:25,238-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:17:33,054-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:17:33,406-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:17:43,414-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:17:53,423-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 10:18:02,556-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 10:18:03,430-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 10:18:09,528-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 10:18:13,438-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:18:23,447-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:18:33,455-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:18:39,874-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:18:43,463-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:18:46,002-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:18:53,471-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:19:03,479-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:19:13,487-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 10:19:17,193-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:19:22,476-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:19:23,496-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:19:33,504-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:19:43,512-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:19:53,520-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:19:54,510-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:19:58,950-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:20:03,528-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:20:13,537-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:20:23,545-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:20:31,828-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:20:33,553-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 10:20:35,425-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 10:20:43,561-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:20:53,569-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:21:03,577-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-08 10:21:09,146-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:21:11,898-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:21:13,585-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:21:23,594-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:21:33,602-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:21:43,610-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:21:46,465-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 10:21:48,372-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 10:21:53,618-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:22:03,627-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:22:13,635-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:22:23,643-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:22:23,783-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:22:24,846-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:22:33,651-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:22:43,660-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:22:53,668-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 10:23:01,102-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 10:23:01,319-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 10:23:03,677-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:23:13,685-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:23:23,694-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:23:33,702-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:23:37,794-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:23:38,419-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:23:43,710-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:23:53,718-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:24:03,727-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:24:13,735-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:24:14,268-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:24:15,737-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:24:23,744-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:24:33,752-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:24:43,761-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 10:24:50,743-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:24:53,055-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:24:53,769-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:25:03,777-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:25:13,786-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:25:23,794-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:25:27,216-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:25:30,373-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:25:33,803-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:25:43,811-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:25:53,819-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:26:03,690-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:26:03,828-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:26:07,691-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:26:13,836-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:26:23,844-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:26:33,853-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 10:26:40,164-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 10:26:43,861-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:26:45,009-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:26:53,870-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:27:03,878-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:27:13,886-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:27:16,638-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:27:22,327-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:27:23,894-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:27:33,902-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:27:43,911-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:27:53,112-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:27:53,919-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:27:59,645-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:28:03,927-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:28:13,935-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:28:23,944-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 10:28:29,587-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 10:28:33,952-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:28:36,963-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:28:43,960-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:28:53,968-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:29:03,976-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 10:29:06,060-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 10:29:13,984-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-08 10:29:14,280-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-No streams
-2013-11-08 10:29:23,993-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:29:34,001-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:29:42,534-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:29:44,009-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:29:51,599-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:29:54,017-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:30:04,025-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:30:14,034-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:30:19,008-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:30:24,042-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:30:28,917-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:30:34,050-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:30:44,058-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:30:54,067-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 10:30:55,482-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 10:31:04,075-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:31:06,235-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:31:14,084-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:31:24,092-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:31:31,956-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:31:34,101-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:31:43,553-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:31:44,109-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:31:54,117-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:32:04,125-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:32:08,430-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:32:14,133-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:32:20,871-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:32:24,141-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:32:34,149-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:32:44,157-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:32:44,904-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:32:54,166-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:32:58,189-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:33:04,174-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:33:14,182-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 10:33:21,378-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 10:33:24,190-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:33:34,198-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:33:35,507-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:33:44,207-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:33:54,215-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:33:57,852-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:34:04,223-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:34:12,825-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:34:14,231-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:34:24,239-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:34:34,248-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 10:34:34,326-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 10:34:44,256-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-08 10:34:50,144-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-No streams
-2013-11-08 10:34:54,264-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:35:04,273-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 10:35:10,802-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 10:35:14,281-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:35:24,290-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:35:27,461-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:35:34,298-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:35:44,307-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:35:47,275-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:35:54,315-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:36:04,323-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:36:04,779-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:36:14,332-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:36:23,749-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:36:24,340-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:36:34,349-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-08 10:36:42,098-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-08 10:36:44,357-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:36:54,365-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:37:00,223-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:37:04,374-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:37:14,382-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:37:19,416-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:37:24,390-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:37:34,398-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:37:36,697-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:37:44,407-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:37:54,415-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:37:56,734-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:38:04,423-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:38:13,171-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:38:14,432-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:38:24,440-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-08 10:38:34,052-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-No streams
-2013-11-08 10:38:34,448-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:38:44,456-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 10:38:49,645-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 10:38:54,464-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:39:04,472-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-08 10:39:11,372-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-No streams
-2013-11-08 10:39:14,481-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:39:24,489-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:39:26,120-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:39:34,498-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:39:44,506-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:39:48,689-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:39:54,515-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:40:02,594-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:40:04,523-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:40:14,531-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:40:24,540-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:40:26,007-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:40:34,548-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-08 10:40:39,068-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-No streams
-2013-11-08 10:40:44,556-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:40:54,564-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:41:03,326-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:41:04,572-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:41:14,580-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:41:15,542-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:41:24,589-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:41:34,597-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 10:41:40,645-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 10:41:44,605-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:41:52,016-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:41:54,614-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:42:04,622-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:42:14,631-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:42:17,962-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:42:24,639-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:42:28,490-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:42:34,647-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:42:44,656-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:42:54,664-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-08 10:42:55,280-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-08 10:43:04,672-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 10:43:04,965-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 10:43:14,680-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:43:24,688-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:43:32,598-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:43:34,697-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 10:43:41,439-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 10:43:44,705-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:43:54,713-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:44:04,721-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:44:09,916-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:44:14,730-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:44:17,912-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:44:24,738-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:44:34,746-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:44:44,754-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:44:47,234-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:44:54,386-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:44:54,763-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:45:04,771-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:45:14,779-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-08 10:45:24,553-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-No streams
-2013-11-08 10:45:24,788-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 10:45:30,860-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 10:45:34,796-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:45:44,804-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:45:54,813-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:46:01,870-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:46:04,821-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:46:07,334-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:46:14,829-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:46:24,838-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:46:34,846-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-08 10:46:39,188-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:46:43,808-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:46:44,854-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:46:54,862-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:47:04,871-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:47:14,879-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-08 10:47:16,507-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:47:20,282-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:47:24,887-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:47:34,895-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:47:44,903-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:47:53,824-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:47:54,911-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:47:56,756-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:48:04,919-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:48:14,928-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:48:24,936-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:48:31,142-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:48:33,230-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:48:34,944-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:48:44,952-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:48:54,961-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:49:04,969-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-08 10:49:08,461-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:49:09,704-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:49:14,977-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:49:24,985-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:49:34,994-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:49:45,002-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:49:45,779-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:49:46,178-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:49:55,011-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:50:05,019-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:50:15,027-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:50:22,652-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:50:23,097-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:50:25,036-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:50:35,044-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:50:45,053-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:50:55,061-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:50:59,126-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:51:00,415-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:51:05,069-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:51:15,077-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:51:25,086-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:51:35,094-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:51:35,600-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:51:37,733-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:51:45,102-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:51:55,111-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:52:05,119-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 10:52:12,074-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:52:15,051-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:52:15,127-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:52:25,135-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:52:35,143-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:52:45,152-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:52:48,549-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-08 10:52:52,369-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-08 10:52:55,160-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:53:05,168-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:53:15,176-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:53:25,023-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:53:25,184-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-08 10:53:29,688-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-No streams
-2013-11-08 10:53:35,196-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:53:45,205-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:53:55,213-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 10:54:01,498-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 10:54:05,221-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:54:07,005-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:54:15,230-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:54:25,238-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:54:35,246-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:54:37,971-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:54:44,323-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:54:45,255-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:54:55,263-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:55:05,271-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:55:14,446-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:55:15,280-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:55:21,641-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:55:25,288-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:55:35,296-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:55:45,304-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:55:50,920-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:55:55,312-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:55:58,959-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:56:05,321-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:56:15,329-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:56:25,338-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:56:27,394-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:56:35,346-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:56:36,277-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:56:45,355-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:56:55,363-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:57:03,868-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:57:05,371-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:57:13,595-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:57:15,380-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:57:25,388-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:57:35,396-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 10:57:40,343-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 10:57:45,404-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:57:50,913-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:57:55,412-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:58:05,420-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:58:15,429-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:58:16,816-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:58:25,437-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-08 10:58:28,231-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-08 10:58:35,446-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:58:45,454-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:58:53,290-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:58:55,462-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:59:05,471-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:59:05,549-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:59:15,479-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:59:25,487-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:59:29,764-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:59:35,495-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 10:59:42,867-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 10:59:45,504-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 10:59:55,512-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:00:05,520-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:00:06,238-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:00:15,529-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-08 11:00:20,185-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-08 11:00:25,537-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:00:35,545-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 11:00:42,713-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 11:00:45,554-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:00:55,562-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-08 11:00:57,504-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-No streams
-2013-11-08 11:01:05,570-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:01:15,578-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:01:19,186-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:01:25,586-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:01:34,821-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:01:35,595-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:01:45,603-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:01:55,611-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:01:55,660-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:02:05,619-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-08 11:02:12,139-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-08 11:02:15,628-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:02:25,636-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:02:32,134-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:02:35,644-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:02:45,652-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-08 11:02:49,458-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-No streams
-2013-11-08 11:02:55,661-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:03:05,669-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:03:08,608-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:03:15,677-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:03:25,686-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:03:26,775-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:03:35,694-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:03:45,082-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:03:45,702-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:03:55,710-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-08 11:04:04,093-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-08 11:04:05,718-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:04:15,727-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:04:21,556-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:04:25,735-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:04:35,744-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-08 11:04:41,412-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-No streams
-2013-11-08 11:04:45,754-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:04:55,763-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:04:58,030-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:05:05,773-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:05:15,782-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-08 11:05:18,730-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-No streams
-2013-11-08 11:05:25,791-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:05:34,504-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:05:35,801-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:05:45,814-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:05:55,816-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:05:56,047-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:06:05,826-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-08 11:06:10,979-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-No streams
-2013-11-08 11:06:15,836-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:06:25,846-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:06:33,365-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:06:35,858-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:06:45,871-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 11:06:47,453-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 11:06:55,890-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:07:05,900-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-08 11:07:10,684-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-No streams
-2013-11-08 11:07:15,916-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 11:07:23,927-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 11:07:25,928-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:07:35,938-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:07:45,947-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:07:48,001-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:07:55,956-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:08:00,400-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:08:05,966-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:08:15,975-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-08 11:08:25,320-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-No streams
-2013-11-08 11:08:25,985-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:08:35,994-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:08:36,874-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:08:46,004-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:08:56,014-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-08 11:09:02,638-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-No streams
-2013-11-08 11:09:06,024-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:09:13,348-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:09:16,034-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:09:26,044-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:09:36,054-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-08 11:09:39,956-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-No streams
-2013-11-08 11:09:46,063-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:09:49,822-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:09:56,072-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:10:06,082-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:10:16,090-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-08 11:10:17,274-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-No streams
-2013-11-08 11:10:26,098-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 11:10:26,297-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 11:10:36,107-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:10:46,115-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 909ms
-2013-11-08 11:10:55,499-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 909ms
-No streams
-2013-11-08 11:10:56,123-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 102ms
-2013-11-08 11:11:02,871-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 102ms
-No streams
-2013-11-08 11:11:06,133-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:11:16,143-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:11:26,155-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:11:31,909-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:11:36,169-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:11:39,245-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:11:46,181-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:11:56,184-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:12:06,200-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:12:09,227-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 11:12:15,719-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 11:12:16,212-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:12:26,226-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:12:36,246-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:12:46,262-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-08 11:12:46,546-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 11:12:52,193-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 11:12:56,274-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:13:06,289-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:13:16,300-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-08 11:13:23,864-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-No streams
-2013-11-08 11:13:26,311-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-2013-11-08 11:13:28,668-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-No streams
-2013-11-08 11:13:36,319-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:13:46,329-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:13:56,339-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 11:14:01,182-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:14:05,140-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:14:06,350-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:14:16,360-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:14:26,371-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:14:36,382-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-08 11:14:38,500-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:14:41,614-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:14:46,393-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:14:56,404-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:15:06,414-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:15:15,817-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:15:16,422-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 11:15:18,089-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 11:15:26,430-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:15:36,438-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:15:46,447-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:15:53,136-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:15:54,562-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:15:56,458-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:16:06,468-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:16:16,478-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:16:26,487-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 11:16:30,454-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:16:31,036-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:16:36,497-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:16:46,507-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:16:56,516-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:17:06,527-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 11:17:07,511-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 11:17:07,772-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 11:17:16,537-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:17:26,545-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:17:36,553-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:17:43,984-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:17:45,089-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:17:46,562-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:17:56,570-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:18:06,582-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:18:16,590-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 11:18:20,458-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:18:22,407-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:18:26,598-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:18:36,607-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:18:46,615-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:18:56,623-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 11:18:56,933-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 11:18:59,726-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 11:19:06,631-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:19:16,640-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:19:26,648-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 11:19:33,407-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 11:19:36,656-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:19:37,043-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:19:46,665-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:19:56,673-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:20:06,681-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:20:09,880-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 11:20:14,362-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 11:20:16,689-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:20:26,697-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:20:36,705-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 11:20:46,355-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 11:20:46,714-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:20:51,679-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:20:56,722-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:21:06,730-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:21:16,738-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:21:22,828-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:21:26,746-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-2013-11-08 11:21:28,997-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 1ms
-No streams
-2013-11-08 11:21:36,754-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:21:46,762-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:21:56,771-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 11:21:59,303-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-08 11:22:06,316-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-No streams
-2013-11-08 11:22:06,779-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:22:16,787-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:22:26,795-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:22:35,776-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:22:36,804-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:22:43,633-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:22:46,812-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:22:56,820-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:23:06,828-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 11:23:12,251-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 11:23:16,836-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:23:20,952-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:23:26,844-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:23:36,853-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:23:46,861-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:23:48,724-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:23:56,869-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:23:58,270-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:24:06,878-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:24:16,886-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:24:25,198-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:24:26,894-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:24:35,588-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:24:36,902-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:24:46,910-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:24:56,918-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:25:01,672-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:25:06,926-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:25:12,906-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:25:16,934-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:25:26,943-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:25:36,951-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 11:25:38,147-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 11:25:46,959-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:25:50,224-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:25:56,967-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:26:06,975-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:26:14,620-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:26:16,984-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:26:26,992-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:26:27,542-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:26:37,000-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:26:47,008-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 11:26:51,094-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 11:26:57,018-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:27:04,860-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:27:07,026-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:27:17,034-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:27:27,042-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:27:27,568-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:27:37,049-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:27:42,178-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:27:47,059-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:27:57,066-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 11:28:04,042-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 11:28:07,074-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:28:17,082-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 11:28:19,497-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 11:28:27,090-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:28:37,098-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 11:28:40,515-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 11:28:47,106-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:28:56,814-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:28:57,114-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:29:07,122-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:29:16,990-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:29:17,130-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:29:27,138-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:29:34,132-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:29:37,146-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:29:47,155-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:29:53,464-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:29:57,163-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:30:07,171-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:30:11,450-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:30:17,179-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:30:27,187-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 11:30:29,939-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 11:30:37,195-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:30:47,203-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:30:48,768-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:30:57,211-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:31:06,412-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:31:07,219-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:31:17,227-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:31:26,087-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:31:27,235-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:31:37,243-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-2013-11-08 11:31:42,888-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-No streams
-2013-11-08 11:31:47,251-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:31:57,259-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:32:03,404-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:32:07,267-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:32:17,275-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:32:19,360-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:32:27,283-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:32:37,291-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 11:32:40,723-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 11:32:47,299-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:32:55,834-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:32:57,307-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:33:07,315-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:33:17,323-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:33:18,040-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:33:27,331-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:33:32,308-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:33:37,339-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:33:47,347-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:33:55,358-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:33:57,355-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:34:07,363-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 11:34:08,782-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 11:34:17,371-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:34:27,379-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:34:32,676-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:34:37,387-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:34:45,256-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:34:47,395-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:34:57,403-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:35:07,411-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:35:09,994-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:35:17,419-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 11:35:21,730-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 11:35:27,427-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:35:37,435-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:35:47,313-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:35:47,443-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:35:57,451-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 11:35:58,205-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 11:36:07,459-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:36:17,467-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:36:24,631-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:36:27,475-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 11:36:34,678-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 11:36:37,483-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:36:47,491-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:36:57,499-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:37:01,949-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:37:07,507-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:37:11,152-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:37:17,515-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:37:27,523-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:37:37,531-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:37:39,267-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:37:47,539-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 11:37:47,627-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 11:37:57,547-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:38:07,555-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:38:16,585-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:38:17,563-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:38:24,100-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:38:27,571-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:38:37,579-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:38:47,587-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 11:38:53,902-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 11:38:57,595-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 11:39:00,574-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 11:39:07,603-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:39:17,611-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:39:27,620-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 11:39:31,222-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 11:39:37,048-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 11:39:37,628-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:39:47,636-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:39:57,643-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:40:07,651-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 11:40:08,540-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:40:13,522-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:40:17,660-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:40:27,667-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:40:37,675-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:40:45,858-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:40:47,683-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 11:40:49,996-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 11:40:57,690-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:41:07,695-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:41:17,703-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 11:41:23,177-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 11:41:26,469-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 11:41:27,711-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:41:37,720-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:41:47,728-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:41:57,736-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:42:00,494-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 11:42:02,944-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 11:42:07,744-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:42:17,752-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:42:27,760-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:42:37,768-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 11:42:37,812-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:42:39,418-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:42:47,776-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:42:57,784-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:43:07,792-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:43:15,130-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:43:15,892-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:43:17,800-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:43:27,807-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:43:37,815-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:43:47,823-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 11:43:52,366-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:43:52,448-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:43:57,831-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:44:07,839-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:44:17,847-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:44:27,855-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:44:28,841-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:44:29,766-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:44:37,863-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:44:47,871-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:44:57,879-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:45:05,316-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:45:07,084-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:45:07,887-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:45:17,894-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:45:27,902-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:45:37,910-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:45:41,790-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:45:44,402-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:45:47,917-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:45:57,925-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:46:07,933-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:46:17,941-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:46:18,264-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 11:46:21,720-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 11:46:27,949-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:46:37,957-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:46:47,965-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 11:46:54,738-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 11:46:57,973-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:46:59,038-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:47:07,980-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:47:17,988-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:47:27,996-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 11:47:31,211-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 11:47:36,357-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 11:47:38,004-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:47:48,011-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:47:58,019-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 11:48:07,686-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 11:48:08,027-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:48:13,674-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:48:18,035-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:48:28,043-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:48:38,051-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:48:44,160-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:48:48,059-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:48:50,993-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:48:58,067-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:49:08,075-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:49:18,083-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:49:20,634-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:49:28,091-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 11:49:28,311-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 11:49:38,099-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:49:48,107-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 11:49:57,107-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 11:49:58,115-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:50:05,629-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:50:08,122-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:50:18,130-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:50:28,138-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 11:50:33,582-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 11:50:38,146-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 11:50:42,948-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 11:50:48,154-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:50:58,162-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:51:08,170-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:51:10,056-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:51:18,178-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:51:20,265-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:51:28,186-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:51:38,194-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 11:51:46,530-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 11:51:48,203-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:51:57,583-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:51:58,211-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:52:08,219-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:52:18,226-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:52:23,004-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:52:28,234-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 11:52:34,901-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 11:52:38,242-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:52:48,250-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:52:58,258-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:52:59,478-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:53:08,266-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 11:53:12,219-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 11:53:18,275-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:53:28,282-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:53:35,952-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:53:38,291-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:53:48,299-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:53:49,538-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:53:58,307-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:54:08,314-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 11:54:12,426-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 11:54:18,322-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 11:54:26,856-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 11:54:28,330-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:54:38,338-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:54:48,345-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:54:48,900-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:54:58,353-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:55:04,173-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:55:08,361-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:55:18,369-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 11:55:25,375-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 11:55:28,377-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:55:38,385-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 11:55:41,492-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 11:55:48,393-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:55:58,401-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 11:56:01,849-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 11:56:08,409-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:56:18,417-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:56:18,809-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:56:28,424-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 11:56:38,322-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 11:56:38,432-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:56:48,440-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:56:56,127-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:56:58,448-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:57:08,456-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:57:14,796-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:57:18,464-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:57:28,472-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 11:57:33,446-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 11:57:38,481-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:57:48,489-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:57:51,270-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:57:58,497-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:58:08,506-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 11:58:10,764-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 11:58:18,514-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 9ms
-2013-11-08 11:58:27,752-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 9ms
-No streams
-2013-11-08 11:58:28,522-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:58:38,530-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:58:48,081-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:58:48,538-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:58:58,546-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:59:04,218-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:59:08,554-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:59:18,561-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 11:59:25,400-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 11:59:28,571-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:59:38,579-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 11:59:40,692-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 11:59:48,587-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 11:59:58,596-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 12:00:02,718-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 12:00:08,604-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:00:17,166-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:00:18,613-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:00:28,621-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:00:38,629-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:00:40,036-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:00:48,638-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:00:53,640-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:00:58,646-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:01:08,654-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:01:17,355-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:01:18,662-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:01:28,671-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 12:01:30,114-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 12:01:38,679-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:01:48,687-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 12:01:54,673-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 12:01:58,695-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 12:02:06,587-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 12:02:08,703-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:02:18,711-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:02:28,720-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:02:31,991-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:02:38,728-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 12:02:43,062-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 12:02:48,736-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:02:58,744-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:03:08,752-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 12:03:09,310-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 12:03:18,760-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:03:19,536-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:03:28,769-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:03:38,778-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:03:46,627-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:03:48,786-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:03:56,010-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:03:58,794-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:04:08,802-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:04:18,811-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:04:23,945-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:04:28,819-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 10ms
-2013-11-08 12:04:32,493-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 10ms
-No streams
-2013-11-08 12:04:38,828-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:04:48,836-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:04:58,844-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 12:05:01,264-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 12:05:08,853-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 12:05:08,959-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 12:05:18,866-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:05:28,874-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 12:05:38,581-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 12:05:38,883-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:05:45,432-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:05:48,891-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:05:58,899-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:06:08,908-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:06:15,900-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:06:18,916-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 12:06:21,906-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 12:06:28,924-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:06:38,934-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:06:48,944-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 12:06:53,219-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:06:58,380-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:06:58,952-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:07:08,962-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:07:18,972-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:07:28,983-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 12:07:30,536-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:07:34,854-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:07:38,993-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:07:49,001-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:07:59,014-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 12:08:07,855-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 12:08:09,024-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:08:11,329-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:08:19,032-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:08:29,046-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:08:39,057-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 12:08:45,173-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:08:47,803-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:08:49,069-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:08:59,081-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:09:09,091-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:09:19,102-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 12:09:22,491-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:09:24,277-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:09:29,112-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:09:39,124-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:09:49,134-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:09:59,145-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 12:09:59,809-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:10:00,751-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:10:09,156-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:10:19,166-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:10:29,176-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:10:37,126-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:10:37,225-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:10:39,187-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:10:49,198-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:10:59,209-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:11:09,220-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 12:11:13,700-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:11:14,444-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:11:19,230-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:11:29,241-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:11:39,252-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:11:49,263-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 12:11:50,175-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 12:11:51,763-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 12:11:59,274-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:12:09,282-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:12:19,290-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:12:26,647-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 12:12:29,080-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 12:12:29,299-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:12:39,307-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:12:49,316-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:12:59,325-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 12:13:03,121-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 12:13:06,398-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 12:13:09,333-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:13:19,342-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:13:29,350-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:13:39,359-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:13:39,595-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 12:13:43,716-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 12:13:49,367-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:13:59,375-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:14:09,384-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 12:14:16,069-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 12:14:19,392-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 12:14:21,035-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 12:14:29,401-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:14:39,409-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:14:49,417-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:14:52,543-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:14:58,352-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:14:59,425-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:15:09,432-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:15:19,440-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 12:15:29,018-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 12:15:29,448-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 12:15:35,670-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 12:15:39,456-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:15:49,464-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:15:59,472-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:16:05,491-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:16:09,479-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:16:12,988-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:16:19,487-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:16:29,495-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:16:39,502-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 12:16:41,965-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 12:16:49,510-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 12:16:50,307-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 12:16:59,518-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:17:09,526-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:17:18,439-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:17:19,534-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 12:17:27,624-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 12:17:29,541-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:17:39,549-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:17:49,557-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 12:17:54,914-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 12:17:59,565-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:18:04,943-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:18:09,573-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:18:19,581-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:18:29,588-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 12:18:31,386-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 12:18:39,597-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 12:18:42,261-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 12:18:49,604-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:18:59,612-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:19:07,861-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:19:09,620-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:19:19,579-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:19:19,627-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:19:29,635-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:19:39,643-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:19:44,335-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:19:49,651-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 12:19:56,898-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 12:19:59,658-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:20:09,666-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:20:19,680-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:20:20,809-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:20:29,688-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:20:34,215-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:20:39,698-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:20:49,706-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:20:57,283-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:20:59,714-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:21:09,723-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 12:21:11,534-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 12:21:19,731-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:21:29,739-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:21:33,757-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:21:39,747-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:21:48,851-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:21:49,755-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:21:59,763-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:22:09,771-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 12:22:10,230-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 12:22:19,779-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:22:26,169-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:22:29,788-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:22:39,796-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:22:46,705-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:22:49,805-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:22:59,813-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:23:03,487-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:23:09,822-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:23:19,830-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 12:23:23,178-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 12:23:29,838-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:23:39,847-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:23:40,805-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:23:49,855-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:23:59,653-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:23:59,864-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:24:09,872-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 12:24:18,124-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 12:24:19,880-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:24:29,888-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 12:24:36,127-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 12:24:39,896-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:24:49,904-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:24:55,441-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:24:59,913-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:25:09,920-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:25:12,602-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:25:19,928-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:25:29,937-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 12:25:32,760-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 12:25:39,945-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:25:49,077-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:25:49,953-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:25:59,961-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:26:09,970-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 12:26:10,077-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 12:26:19,978-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:26:25,551-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:26:29,987-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:26:39,995-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:26:47,395-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:26:50,003-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:27:00,012-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:27:02,026-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:27:10,020-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:27:20,029-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 12:27:24,714-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 12:27:30,038-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 12:27:38,500-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 12:27:40,046-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:27:50,054-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:28:00,062-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:28:02,031-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:28:10,069-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 12:28:14,974-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 12:28:20,077-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:28:30,085-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:28:39,350-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:28:40,095-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:28:50,102-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:28:51,449-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:29:00,110-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:29:10,118-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:29:16,668-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:29:20,126-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:29:27,923-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:29:30,134-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:29:40,142-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:29:50,150-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:29:53,986-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:30:00,159-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 12:30:04,397-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 12:30:10,167-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:30:20,175-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:30:30,184-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:30:31,305-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:30:40,192-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:30:40,871-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:30:50,200-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:31:00,208-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 12:31:08,623-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 12:31:10,217-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 12:31:17,345-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 12:31:20,225-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:31:30,233-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:31:40,241-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:31:45,941-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:31:50,249-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 12:31:53,819-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 12:32:00,257-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:32:10,264-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:32:20,272-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 12:32:23,260-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 12:32:30,280-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 12:32:30,293-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 12:32:40,288-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:32:50,296-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:33:00,304-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:33:00,577-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 12:33:06,766-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 12:33:10,312-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:33:20,320-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:33:30,327-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 12:33:37,896-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 12:33:40,335-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 12:33:43,241-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 12:33:50,343-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:34:00,351-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:34:10,359-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:34:15,213-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 12:34:19,715-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 12:34:20,366-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:34:30,374-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:34:40,382-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:34:50,390-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:34:52,531-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:34:56,189-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:35:00,398-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:35:10,406-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:35:20,413-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 12:35:29,850-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 12:35:30,421-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:35:32,663-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:35:40,429-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:35:50,437-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:36:00,445-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:36:07,167-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:36:09,137-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:36:10,453-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:36:20,461-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:36:30,469-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:36:40,478-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 12:36:44,486-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:36:45,611-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:36:50,486-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:37:00,494-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:37:10,502-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:37:20,510-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:37:21,803-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:37:22,085-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:37:30,519-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:37:40,527-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:37:50,535-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 12:37:58,559-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 12:37:59,120-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 12:38:00,543-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:38:10,551-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:38:20,559-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:38:30,568-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:38:35,033-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 12:38:36,439-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 12:38:40,576-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:38:50,584-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:39:00,592-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:39:10,601-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:39:11,507-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:39:13,757-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:39:20,609-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:39:30,617-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:39:40,625-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:39:47,981-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:39:50,633-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 12:39:51,076-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 12:40:00,641-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:40:10,649-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:40:20,657-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 12:40:24,456-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:40:28,393-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:40:30,665-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:40:40,673-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:40:50,681-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:41:00,689-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:41:00,929-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 12:41:05,712-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 12:41:10,697-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:41:20,705-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:41:30,713-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 12:41:37,403-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 12:41:40,721-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 12:41:43,029-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 12:41:50,730-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:42:00,737-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:42:10,746-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:42:13,877-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:42:20,347-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:42:20,754-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:42:30,762-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:42:40,769-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:42:50,351-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:42:50,777-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 12:42:57,665-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 12:43:00,785-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:43:10,793-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:43:20,801-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:43:26,825-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:43:30,809-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:43:34,983-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:43:40,817-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:43:50,825-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:44:00,833-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:44:03,299-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:44:10,841-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 12:44:12,301-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 12:44:20,849-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:44:30,856-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:44:39,774-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:44:40,865-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:44:49,619-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:44:50,872-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:45:00,880-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:45:10,888-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:45:16,248-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:45:20,896-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 12:45:26,938-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 12:45:30,904-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:45:40,912-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:45:50,920-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:45:52,722-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:46:00,928-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:46:04,255-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:46:10,937-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:46:20,945-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:46:29,196-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:46:30,953-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:46:40,961-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:46:41,573-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:46:50,970-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:47:00,978-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-08 12:47:05,670-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-No streams
-2013-11-08 12:47:10,986-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:47:18,891-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:47:20,995-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:47:31,003-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:47:41,011-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 12:47:42,146-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 12:47:51,019-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:47:56,209-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:48:01,028-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:48:11,036-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 12:48:18,619-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 12:48:21,045-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:48:31,053-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 12:48:33,527-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 12:48:41,061-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:48:51,070-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:48:55,093-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:49:01,078-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:49:10,845-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:49:11,086-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:49:21,094-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:49:31,103-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:49:31,567-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:49:41,111-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 12:49:48,164-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 12:49:51,120-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:50:01,128-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:50:08,041-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:50:11,136-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:50:21,145-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:50:25,481-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:50:31,153-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:50:41,161-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 12:50:44,515-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 12:50:51,170-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:51:01,178-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:51:02,799-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:51:11,186-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:51:20,989-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:51:21,195-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:51:31,203-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 12:51:40,117-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 12:51:41,211-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:51:51,219-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:51:57,463-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:52:01,227-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:52:11,235-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:52:17,435-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:52:21,244-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:52:31,252-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:52:33,937-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:52:41,260-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:52:51,268-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 12:52:54,754-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 12:53:01,276-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:53:10,411-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:53:11,284-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:53:21,292-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:53:31,301-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:53:32,071-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:53:41,309-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:53:46,885-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:53:51,318-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:54:01,326-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 12:54:09,390-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 12:54:11,334-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:54:21,343-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 12:54:23,360-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 12:54:31,351-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:54:41,359-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 12:54:46,707-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 12:54:51,367-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-2013-11-08 12:54:59,833-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 0ms
-No streams
-2013-11-08 12:55:01,376-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:55:11,384-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:55:21,392-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 12:55:24,026-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 12:55:31,401-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:55:36,308-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:55:41,409-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:55:51,417-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 12:56:01,343-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 12:56:01,426-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:56:11,434-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 12:56:12,782-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 12:56:21,442-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:56:31,451-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:56:38,661-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:56:41,459-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:56:49,256-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:56:51,467-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:57:01,475-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:57:11,486-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:57:15,980-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:57:21,494-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:57:25,730-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:57:31,502-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:57:41,511-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:57:51,519-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:57:53,298-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:58:01,527-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:58:02,204-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:58:11,536-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:58:21,544-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 12:58:30,617-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 12:58:31,552-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 12:58:38,679-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 12:58:41,561-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:58:51,571-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:59:01,580-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 12:59:07,935-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 12:59:11,591-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 12:59:15,153-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 12:59:21,602-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:59:31,613-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 12:59:41,624-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 12:59:45,253-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 12:59:51,627-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 12:59:51,635-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:00:01,645-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:00:11,657-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:00:21,667-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 3ms
-2013-11-08 13:00:22,572-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 2ms, rtt: 3ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 13:00:28,101-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 13:00:31,677-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:00:41,690-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:00:51,708-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-2013-11-08 13:00:59,890-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 3ms
-No streams
-2013-11-08 13:01:01,712-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-08 13:01:04,575-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-No streams
-2013-11-08 13:01:11,716-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:01:21,722-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:01:31,727-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 13:01:37,207-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-08 13:01:41,049-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-No streams
-2013-11-08 13:01:41,735-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:01:51,755-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:02:01,767-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:02:11,775-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 13:02:14,524-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 13:02:17,523-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 13:02:21,783-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:02:31,793-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:02:41,802-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:02:51,812-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 13:02:51,843-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 13:02:53,996-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 13:03:01,821-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:03:11,830-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:03:21,838-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 13:03:29,160-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 13:03:30,471-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 13:03:31,847-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:03:41,855-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:03:51,865-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:04:01,876-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 13:04:06,479-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 13:04:06,944-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 13:04:11,886-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:04:21,897-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:04:31,906-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:04:41,914-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 13:04:43,418-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 13:04:43,796-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 13:04:51,922-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:05:01,929-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:05:11,937-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 13:05:19,893-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 13:05:21,114-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 13:05:21,945-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:05:31,954-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:05:41,966-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:05:51,976-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 13:05:56,368-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 13:05:58,432-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 13:06:01,988-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:06:12,000-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:06:22,010-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:06:32,018-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 13:06:32,842-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:06:35,750-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 13:06:42,026-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:06:52,034-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:07:02,044-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 13:07:09,316-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 13:07:12,052-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:07:13,068-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 13:07:22,060-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:07:32,068-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:07:42,076-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 13:07:45,789-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 13:07:50,386-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 13:07:52,085-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:08:02,093-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:08:12,102-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:08:22,110-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 13:08:22,264-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 13:08:27,705-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 13:08:32,119-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:08:42,127-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:08:52,136-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 13:08:58,737-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 13:09:02,144-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 13:09:05,023-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 13:09:12,153-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:09:22,166-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:09:32,176-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 13:09:35,212-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 13:09:42,186-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 13:09:42,341-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 13:09:52,196-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:10:02,207-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 13:10:11,686-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 13:10:12,216-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 134ms
-2013-11-08 13:10:19,791-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 134ms
-No streams
-2013-11-08 13:10:22,226-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:10:32,236-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:10:42,244-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 13:10:48,160-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 13:10:52,257-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 13:10:56,977-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 13:11:02,269-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:11:12,279-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:11:22,289-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 13:11:24,633-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 13:11:32,298-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:11:34,294-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-08 13:11:38,425-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:34802] for addition
-2013-11-08 13:11:38,425-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:34802] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:34802]
-2013-11-08 13:11:38,425-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:34802]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f35
-2013-11-08 13:11:38,460-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f35
-MetaChannel [context: cps-34802, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:34802]
-2013-11-08 13:11:38,460-0600 INFO MetaChannel MetaChannel [context: cps-34802, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:34802]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-08 13:11:38,460-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-08 13:11:38,501-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-08 13:11:38,501-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-08 13:11:38,540-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:34803] for addition
-2013-11-08 13:11:38,540-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:34803] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:34803]
-2013-11-08 13:11:38,540-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:34803]
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f34
-2013-11-08 13:11:38,541-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f34
-MetaChannel [context: cps-34803, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:34803]
-2013-11-08 13:11:38,541-0600 INFO MetaChannel MetaChannel [context: cps-34803, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:34803]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-08 13:11:38,542-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-08 13:11:38,582-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-08 13:11:38,582-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-08 13:11:38,596-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 137.0) submitting urn:job-0-1383759275050-1383759275051: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-08 13:11:38,596-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 137.0) submitting urn:job-0-1383759275050-1383759275051: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-08 13:11:38,597-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-08 13:11:38,597-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-08 13:11:38,604-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7606, JOBSTATUS) unregistering (send)
-2013-11-08 13:11:38,604-0600 INFO RequestHandler Handler(tag: 7606, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-08 13:11:38,626-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:34804] for addition
-2013-11-08 13:11:38,626-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:34804] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:34804]
-2013-11-08 13:11:38,626-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:34804]
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-08 13:11:38,663-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f33
-2013-11-08 13:11:38,663-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f33
-MetaChannel [context: cps-34804, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:34804]
-2013-11-08 13:11:38,663-0600 INFO MetaChannel MetaChannel [context: cps-34804, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:34804]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-08 13:11:38,663-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-08 13:11:38,697-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-crush.mcs.anl.gov:0 (quality: 52.0) submitting urn:job-1-1383759275053-1383759275054: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-08 13:11:38,697-0600 INFO Cpu crush.mcs.anl.gov:0 (quality: 52.0) submitting urn:job-1-1383759275053-1383759275054: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-08 13:11:38,704-0600 DEBUG Cpu Active workerid=crush.mcs.anl.gov:000000
-Handler(tag: 135, JOBSTATUS) unregistering (send)
-2013-11-08 13:11:38,704-0600 INFO RequestHandler Handler(tag: 135, JOBSTATUS) unregistering (send)
-Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-08 13:11:38,704-0600 INFO RequestHandler Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-08 13:11:38,765-0600 INFO RequestHandler Handler(tag: 42566, SUBMITJOB) unregistering (send)
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 13:11:38,765-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel config: null -> {keepalive=null}
-2013-11-08 13:11:38,765-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:34804] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 13:11:38,765-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:34804] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-08 13:11:38,766-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-34804, boundTo: TCPChannel [type: server, contact: 127.0.0.1:34804]] binding to NullChannel
-2013-11-08 13:11:38,766-0600 INFO MetaChannel MetaChannel [context: cps-34804, boundTo: TCPChannel [type: server, contact: 127.0.0.1:34804]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:34804]
-2013-11-08 13:11:38,766-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:34804]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 13:11:38,766-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-08 13:11:38,766-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-08 13:11:38,785-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-08 13:11:38,785-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-08 13:11:38,785-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:34804] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-08 13:11:38,785-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:34804] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-08 13:11:38,785-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:34804]
-2013-11-08 13:11:38,786-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:34804]
-2013-11-08 13:11:40,631-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-08 13:11:40,631-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-08 13:11:40,631-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 13:11:40,631-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 7607, JOBSTATUS) unregistering (send)
-2013-11-08 13:11:40,631-0600 INFO RequestHandler Handler(tag: 7607, JOBSTATUS) unregistering (send)
-2013-11-08 13:11:40,631-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-Channel config: null -> {keepalive=null}
-2013-11-08 13:11:40,631-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:34802] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 13:11:40,632-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:34802] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-grind.mcs.anl.gov:0 (quality: 138.0) submitting urn:job-2-1383759275056-1383759275057: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-08 13:11:40,632-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 138.0) submitting urn:job-2-1383759275056-1383759275057: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-null: Channel shut down
-2013-11-08 13:11:40,632-0600 INFO AbstractTCPChannel null: Channel shut down
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.submit(Cpu.java:281)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.launchSequential(Cpu.java:244)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.launch(Cpu.java:229)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.pull(Cpu.java:203)
- at org.globus.cog.abstraction.coaster.service.job.manager.PullThread.run(PullThread.java:131)
-2013-11-08 13:11:40,632-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.submit(Cpu.java:281)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.launchSequential(Cpu.java:244)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.launch(Cpu.java:229)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.pull(Cpu.java:203)
- at org.globus.cog.abstraction.coaster.service.job.manager.PullThread.run(PullThread.java:131)
-MetaChannel [context: cps-34802, boundTo: TCPChannel [type: server, contact: 127.0.0.1:34802]] binding to NullChannel
-2013-11-08 13:11:40,632-0600 INFO MetaChannel MetaChannel [context: cps-34802, boundTo: TCPChannel [type: server, contact: 127.0.0.1:34802]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:34802]
-2013-11-08 13:11:40,632-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:34802]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 13:11:40,632-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-08 13:11:40,632-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 13:11:40,632-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-08 13:11:40,632-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 13:11:40,633-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-08 13:11:40,633-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 13:11:40,633-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-08 13:11:40,633-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 13:11:40,633-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-08 13:11:40,633-0600 INFO ChannelManager Channel exception handled
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-08 13:11:40,637-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-08 13:11:40,638-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-08 13:11:40,638-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7608, JOBSTATUS) unregistering (send)
-2013-11-08 13:11:40,638-0600 INFO RequestHandler Handler(tag: 7608, JOBSTATUS) unregistering (send)
-Channel is closed. Removing.
-2013-11-08 13:11:40,651-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-08 13:11:40,651-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-08 13:11:40,651-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:34802] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-08 13:11:40,651-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:34802] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-08 13:11:40,652-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:34802]
-2013-11-08 13:11:40,652-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:34802]
-2013-11-08 13:11:40,885-0600 DEBUG Cpu Completed
-crush.mcs.anl.gov:0 jobTerminated
-2013-11-08 13:11:40,886-0600 INFO Cpu crush.mcs.anl.gov:0 jobTerminated
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 13:11:40,886-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 13:11:40,886-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-Channel config: null -> {keepalive=null}
-2013-11-08 13:11:40,886-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-08 13:11:40,886-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-Handler(tag: 136, JOBSTATUS) unregistering (send)
-2013-11-08 13:11:40,886-0600 INFO RequestHandler Handler(tag: 136, JOBSTATUS) unregistering (send)
-TCPChannel [type: server, contact: 127.0.0.1:34803] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 13:11:40,886-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:34803] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 13:11:40,886-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-null: Channel shut down
-2013-11-08 13:11:40,886-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-34803, boundTo: TCPChannel [type: server, contact: 127.0.0.1:34803]] binding to NullChannel
-2013-11-08 13:11:40,886-0600 INFO MetaChannel MetaChannel [context: cps-34803, boundTo: TCPChannel [type: server, contact: 127.0.0.1:34803]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:34803]
-2013-11-08 13:11:40,886-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:34803]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 13:11:40,886-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-08 13:11:40,887-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 13:11:40,887-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-08 13:11:40,887-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 13:11:40,887-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-08 13:11:40,887-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 13:11:40,887-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-08 13:11:40,887-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 13:11:40,887-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-08 13:11:40,887-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-08 13:11:40,892-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-08 13:11:40,892-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-08 13:11:40,892-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:34803] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-08 13:11:40,892-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:34803] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-08 13:11:40,893-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:34803]
-2013-11-08 13:11:40,893-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:34803]
-No streams
-2013-11-08 13:11:42,317-0600 INFO Multiplexer No streams
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-08 13:11:42,653-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-08 13:11:42,653-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-08 13:11:42,653-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-08 13:11:42,653-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-08 13:11:42,653-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-Handler(tag: 7609, JOBSTATUS) unregistering (send)
-2013-11-08 13:11:42,653-0600 INFO RequestHandler Handler(tag: 7609, JOBSTATUS) unregistering (send)
-2013-11-08 13:11:42,653-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-No streams
-2013-11-08 13:11:52,326-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 13:12:01,107-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 13:12:02,335-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 13:12:11,613-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 13:12:12,344-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:12:22,353-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:12:32,361-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 13:12:37,581-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 13:12:42,372-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 13:12:48,931-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 13:12:52,385-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:13:02,396-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:13:12,412-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 13:13:14,055-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 13:13:22,430-0600 INFO Multiplexer No streams
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-08 13:13:25,081-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:34818] for addition
-2013-11-08 13:13:25,082-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:34818] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:34818]
-2013-11-08 13:13:25,082-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:34818]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f32
-2013-11-08 13:13:25,105-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f32
-MetaChannel [context: cps-34818, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:34818]
-2013-11-08 13:13:25,105-0600 INFO MetaChannel MetaChannel [context: cps-34818, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:34818]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-08 13:13:25,105-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-08 13:13:25,147-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-08 13:13:25,147-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-08 13:13:25,188-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:34819] for addition
-2013-11-08 13:13:25,188-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:34819] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:34819]
-2013-11-08 13:13:25,188-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:34819]
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f31
-2013-11-08 13:13:25,211-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f31
-MetaChannel [context: cps-34819, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:34819]
-2013-11-08 13:13:25,211-0600 INFO MetaChannel MetaChannel [context: cps-34819, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:34819]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-08 13:13:25,211-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-08 13:13:25,231-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-08 13:13:25,248-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 139.0) submitting urn:job-0-1383759275059-1383759275060: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-08 13:13:25,248-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 139.0) submitting urn:job-0-1383759275059-1383759275060: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-08 13:13:25,249-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-08 13:13:25,249-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-08 13:13:25,252-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-08 13:13:25,254-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7610, JOBSTATUS) unregistering (send)
-2013-11-08 13:13:25,254-0600 INFO RequestHandler Handler(tag: 7610, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-08 13:13:25,292-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:34820] for addition
-2013-11-08 13:13:25,292-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:34820] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:34820]
-2013-11-08 13:13:25,292-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:34820]
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f30
-2013-11-08 13:13:25,312-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f30
-MetaChannel [context: cps-34820, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:34820]
-2013-11-08 13:13:25,313-0600 INFO MetaChannel MetaChannel [context: cps-34820, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:34820]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-08 13:13:25,313-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-08 13:13:25,333-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-08 13:13:25,349-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-crush.mcs.anl.gov:0 (quality: 53.0) submitting urn:job-1-1383759275062-1383759275063: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-08 13:13:25,349-0600 INFO Cpu crush.mcs.anl.gov:0 (quality: 53.0) submitting urn:job-1-1383759275062-1383759275063: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-08 13:13:25,353-0600 INFO RequestHandler Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-08 13:13:25,355-0600 DEBUG Cpu Active workerid=crush.mcs.anl.gov:000000
-Handler(tag: 137, JOBSTATUS) unregistering (send)
-2013-11-08 13:13:25,356-0600 INFO RequestHandler Handler(tag: 137, JOBSTATUS) unregistering (send)
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 13:13:26,248-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 13:13:27,276-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-08 13:13:27,277-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-08 13:13:27,277-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 13:13:27,277-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 7611, JOBSTATUS) unregistering (send)
-2013-11-08 13:13:27,277-0600 INFO RequestHandler Handler(tag: 7611, JOBSTATUS) unregistering (send)
-Channel config: null -> {keepalive=null}
-2013-11-08 13:13:27,277-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-08 13:13:27,277-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-TCPChannel [type: server, contact: 127.0.0.1:34818] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 13:13:27,277-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:34818] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 13:13:27,277-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-null: Channel shut down
-2013-11-08 13:13:27,277-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-34818, boundTo: TCPChannel [type: server, contact: 127.0.0.1:34818]] binding to NullChannel
-2013-11-08 13:13:27,277-0600 INFO MetaChannel MetaChannel [context: cps-34818, boundTo: TCPChannel [type: server, contact: 127.0.0.1:34818]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:34818]
-2013-11-08 13:13:27,278-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:34818]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 13:13:27,278-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-08 13:13:27,278-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 13:13:27,278-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-08 13:13:27,278-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 13:13:27,278-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-08 13:13:27,278-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 13:13:27,278-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-08 13:13:27,278-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 13:13:27,278-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-08 13:13:27,278-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-08 13:13:27,280-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-08 13:13:27,280-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-08 13:13:27,280-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:34818] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-08 13:13:27,280-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:34818] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-08 13:13:27,280-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:34818]
-2013-11-08 13:13:27,280-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:34818]
-2013-11-08 13:13:27,380-0600 DEBUG Cpu Completed
-crush.mcs.anl.gov:0 jobTerminated
-2013-11-08 13:13:27,380-0600 INFO Cpu crush.mcs.anl.gov:0 jobTerminated
-2013-11-08 13:13:27,380-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 13:13:27,380-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 138, JOBSTATUS) unregistering (send)
-2013-11-08 13:13:27,380-0600 INFO RequestHandler Handler(tag: 138, JOBSTATUS) unregistering (send)
-Channel config: null -> {}
-2013-11-08 13:13:27,381-0600 INFO AbstractCoasterChannel Channel config: null -> {}
-TCPChannel [type: server, contact: 127.0.0.1:34819] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 13:13:27,381-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:34819] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 13:13:27,381-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 2
-null: Channel shut down
-2013-11-08 13:13:27,381-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-34819, boundTo: TCPChannel [type: server, contact: 127.0.0.1:34819]] binding to NullChannel
-2013-11-08 13:13:27,381-0600 INFO MetaChannel MetaChannel [context: cps-34819, boundTo: TCPChannel [type: server, contact: 127.0.0.1:34819]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:34819]
-2013-11-08 13:13:27,381-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:34819]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 13:13:27,381-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-08 13:13:27,382-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 13:13:27,382-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-08 13:13:27,382-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 13:13:27,382-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-08 13:13:27,382-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 13:13:27,382-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-08 13:13:27,382-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 13:13:27,382-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-08 13:13:27,382-0600 INFO ChannelManager Channel exception handled
-2013-11-08 13:13:27,382-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-Channel is closed. Removing.
-2013-11-08 13:13:27,400-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-08 13:13:27,400-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {}
-2013-11-08 13:13:27,400-0600 INFO AbstractCoasterChannel Channel config: null -> {}
-TCPChannel [type: server, contact: 127.0.0.1:34819] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-08 13:13:27,400-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:34819] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-08 13:13:27,400-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:34819]
-2013-11-08 13:13:27,401-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:34819]
-Avg stream buf: 0
-2013-11-08 13:13:32,440-0600 INFO Multiplexer Avg stream buf: 0
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-131138.582, now=131108-131339.425, channel=TCPChannel [type: server, contact: 127.0.0.1:34802]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 13:13:39,425-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-131138.582, now=131108-131339.425, channel=TCPChannel [type: server, contact: 127.0.0.1:34802]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-08 13:13:39,426-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-131138.582, now=131108-131339.425, channel=TCPChannel [type: server, contact: 127.0.0.1:34802]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 13:13:39,426-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-131138.582, now=131108-131339.425, channel=TCPChannel [type: server, contact: 127.0.0.1:34802]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-08 13:13:39,426-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-131138.582, now=131108-131339.425, channel=TCPChannel [type: server, contact: 127.0.0.1:34802]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 13:13:39,426-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-131138.582, now=131108-131339.425, channel=TCPChannel [type: server, contact: 127.0.0.1:34802]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-08 13:13:39,426-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-131138.582, now=131108-131339.425, channel=TCPChannel [type: server, contact: 127.0.0.1:34802]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 13:13:39,426-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-131138.582, now=131108-131339.425, channel=TCPChannel [type: server, contact: 127.0.0.1:34802]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-08 13:13:39,426-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-131138.582, now=131108-131339.425, channel=TCPChannel [type: server, contact: 127.0.0.1:34802]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 13:13:39,426-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-131138.582, now=131108-131339.425, channel=TCPChannel [type: server, contact: 127.0.0.1:34802]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-08 13:13:39,426-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:34802] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-131138.582, now=131108-131339.425, channel=TCPChannel [type: server, contact: 127.0.0.1:34802]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 13:13:39,426-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:34802] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-131138.582, now=131108-131339.425, channel=TCPChannel [type: server, contact: 127.0.0.1:34802]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-08 13:13:39,426-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-131138.663, now=131108-131339.541, channel=TCPChannel [type: server, contact: 127.0.0.1:34803]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 13:13:39,541-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-131138.663, now=131108-131339.541, channel=TCPChannel [type: server, contact: 127.0.0.1:34803]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-08 13:13:39,541-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-131138.663, now=131108-131339.541, channel=TCPChannel [type: server, contact: 127.0.0.1:34803]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 13:13:39,541-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-131138.663, now=131108-131339.541, channel=TCPChannel [type: server, contact: 127.0.0.1:34803]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-08 13:13:39,541-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-131138.663, now=131108-131339.541, channel=TCPChannel [type: server, contact: 127.0.0.1:34803]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 13:13:39,541-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-131138.663, now=131108-131339.541, channel=TCPChannel [type: server, contact: 127.0.0.1:34803]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-08 13:13:39,541-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-131138.663, now=131108-131339.541, channel=TCPChannel [type: server, contact: 127.0.0.1:34803]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 13:13:39,541-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-131138.663, now=131108-131339.541, channel=TCPChannel [type: server, contact: 127.0.0.1:34803]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-08 13:13:39,541-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-131138.663, now=131108-131339.541, channel=TCPChannel [type: server, contact: 127.0.0.1:34803]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 13:13:39,541-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-131138.663, now=131108-131339.541, channel=TCPChannel [type: server, contact: 127.0.0.1:34803]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-08 13:13:39,541-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:34803] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-131138.663, now=131108-131339.541, channel=TCPChannel [type: server, contact: 127.0.0.1:34803]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 13:13:39,541-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:34803] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-131138.663, now=131108-131339.541, channel=TCPChannel [type: server, contact: 127.0.0.1:34803]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-08 13:13:39,541-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-131138.765, now=131108-131339.627, channel=TCPChannel [type: server, contact: 127.0.0.1:34804]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 13:13:39,627-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-131138.765, now=131108-131339.627, channel=TCPChannel [type: server, contact: 127.0.0.1:34804]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-08 13:13:39,627-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:34804] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-131138.765, now=131108-131339.627, channel=TCPChannel [type: server, contact: 127.0.0.1:34804]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 13:13:39,627-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:34804] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-131138.765, now=131108-131339.627, channel=TCPChannel [type: server, contact: 127.0.0.1:34804]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-08 13:13:39,627-0600 INFO ChannelManager Channel already shut down
-Avg stream buf: 0
-2013-11-08 13:13:42,452-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:13:50,529-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-08 13:13:52,454-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-08 13:14:02,466-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 13:14:03,567-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-Avg stream buf: 0
-2013-11-08 13:14:12,474-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-08 13:14:22,494-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 13:14:27,003-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-08 13:14:32,495-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 13:14:40,885-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-Avg stream buf: 0
-2013-11-08 13:14:42,513-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-08 13:14:52,530-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-08 13:15:02,546-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-2013-11-08 13:15:03,478-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 2ms, rtt: 2ms
-Avg stream buf: 0
-2013-11-08 13:15:12,562-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 3ms, rtt: 4ms
-2013-11-08 13:15:18,205-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 3ms, rtt: 4ms
-Avg stream buf: 0
-2013-11-08 13:15:22,576-0600 INFO Multiplexer Avg stream buf: 0
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-131325.231, now=131108-131526.082, channel=TCPChannel [type: server, contact: 127.0.0.1:34818]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 13:15:26,082-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-131325.231, now=131108-131526.082, channel=TCPChannel [type: server, contact: 127.0.0.1:34818]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-08 13:15:26,083-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-131325.231, now=131108-131526.082, channel=TCPChannel [type: server, contact: 127.0.0.1:34818]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 13:15:26,083-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-131325.231, now=131108-131526.082, channel=TCPChannel [type: server, contact: 127.0.0.1:34818]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-08 13:15:26,083-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-131325.231, now=131108-131526.082, channel=TCPChannel [type: server, contact: 127.0.0.1:34818]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 13:15:26,083-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-131325.231, now=131108-131526.082, channel=TCPChannel [type: server, contact: 127.0.0.1:34818]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-08 13:15:26,083-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-131325.231, now=131108-131526.082, channel=TCPChannel [type: server, contact: 127.0.0.1:34818]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 13:15:26,083-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-131325.231, now=131108-131526.082, channel=TCPChannel [type: server, contact: 127.0.0.1:34818]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-08 13:15:26,083-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-131325.231, now=131108-131526.082, channel=TCPChannel [type: server, contact: 127.0.0.1:34818]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 13:15:26,083-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-131325.231, now=131108-131526.082, channel=TCPChannel [type: server, contact: 127.0.0.1:34818]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-08 13:15:26,083-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:34818] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-131325.231, now=131108-131526.082, channel=TCPChannel [type: server, contact: 127.0.0.1:34818]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 13:15:26,083-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:34818] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-131325.231, now=131108-131526.082, channel=TCPChannel [type: server, contact: 127.0.0.1:34818]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-08 13:15:26,083-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-131325.333, now=131108-131526.188, channel=TCPChannel [type: server, contact: 127.0.0.1:34819]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 13:15:26,188-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-131325.333, now=131108-131526.188, channel=TCPChannel [type: server, contact: 127.0.0.1:34819]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-08 13:15:26,189-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-131325.333, now=131108-131526.188, channel=TCPChannel [type: server, contact: 127.0.0.1:34819]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 13:15:26,189-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-131325.333, now=131108-131526.188, channel=TCPChannel [type: server, contact: 127.0.0.1:34819]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-08 13:15:26,189-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-131325.333, now=131108-131526.188, channel=TCPChannel [type: server, contact: 127.0.0.1:34819]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 13:15:26,189-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-131325.333, now=131108-131526.188, channel=TCPChannel [type: server, contact: 127.0.0.1:34819]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-08 13:15:26,189-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-131325.333, now=131108-131526.188, channel=TCPChannel [type: server, contact: 127.0.0.1:34819]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 13:15:26,189-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-131325.333, now=131108-131526.188, channel=TCPChannel [type: server, contact: 127.0.0.1:34819]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-08 13:15:26,189-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-131325.333, now=131108-131526.188, channel=TCPChannel [type: server, contact: 127.0.0.1:34819]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 13:15:26,189-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-131325.333, now=131108-131526.188, channel=TCPChannel [type: server, contact: 127.0.0.1:34819]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {}
-2013-11-08 13:15:26,189-0600 INFO AbstractCoasterChannel Channel config: null -> {}
-TCPChannel [type: server, contact: 127.0.0.1:34819] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-131325.333, now=131108-131526.188, channel=TCPChannel [type: server, contact: 127.0.0.1:34819]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 13:15:26,189-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:34819] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-131325.333, now=131108-131526.188, channel=TCPChannel [type: server, contact: 127.0.0.1:34819]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-08 13:15:26,189-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-131325.353, now=131108-131526.292, channel=TCPChannel [type: server, contact: 127.0.0.1:34820]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 13:15:26,292-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-131325.353, now=131108-131526.292, channel=TCPChannel [type: server, contact: 127.0.0.1:34820]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-08 13:15:26,293-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:34820] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-131325.353, now=131108-131526.292, channel=TCPChannel [type: server, contact: 127.0.0.1:34820]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 13:15:26,293-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:34820] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-131325.353, now=131108-131526.292, channel=TCPChannel [type: server, contact: 127.0.0.1:34820]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-null: Channel shut down
-2013-11-08 13:15:26,293-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-34820, boundTo: TCPChannel [type: server, contact: 127.0.0.1:34820]] binding to NullChannel
-2013-11-08 13:15:26,293-0600 INFO MetaChannel MetaChannel [context: cps-34820, boundTo: TCPChannel [type: server, contact: 127.0.0.1:34820]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:34820]
-2013-11-08 13:15:26,293-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:34820]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-131325.353, now=131108-131526.292, channel=TCPChannel [type: server, contact: 127.0.0.1:34820]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 13:15:26,293-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-131325.353, now=131108-131526.292, channel=TCPChannel [type: server, contact: 127.0.0.1:34820]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel exception handled
-2013-11-08 13:15:26,293-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-08 13:15:26,309-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-08 13:15:26,309-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-08 13:15:26,309-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:34820] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-08 13:15:26,309-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:34820] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-08 13:15:26,309-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:34820]
-2013-11-08 13:15:26,309-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:34820]
-No streams
-2013-11-08 13:15:32,590-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 13:15:39,952-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 13:15:42,600-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:15:52,610-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 13:15:55,521-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 13:16:02,623-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:16:12,635-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 13:16:16,426-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 13:16:22,646-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:16:32,659-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 13:16:32,839-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 13:16:42,669-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:16:52,680-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 13:16:52,901-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 13:17:02,691-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 13:17:10,157-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 13:17:12,706-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:17:22,718-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 13:17:29,375-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 13:17:32,729-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:17:42,741-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 13:17:47,475-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 13:17:52,752-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:18:02,762-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 13:18:05,848-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 13:18:12,774-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:18:22,784-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 13:18:24,793-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 13:18:32,796-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 13:18:42,322-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 13:18:42,804-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:18:52,815-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 3ms
-2013-11-08 13:19:02,112-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 3ms
-No streams
-2013-11-08 13:19:02,824-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:19:12,836-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 13:19:18,797-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 13:19:22,847-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:19:32,857-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 13:19:39,429-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 13:19:42,868-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:19:52,882-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:19:55,270-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 13:20:02,895-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:20:12,906-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:20:16,746-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 13:20:22,916-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 13:20:31,745-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 13:20:32,926-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:20:42,934-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:20:52,943-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:20:54,064-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 13:21:02,951-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 13:21:08,218-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 13:21:12,959-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:21:22,967-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:21:31,382-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 13:21:32,975-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:21:42,985-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 13:21:44,692-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 13:21:52,995-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:22:03,005-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 13:22:08,702-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 13:22:13,022-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 13:22:21,166-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 13:22:23,026-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:22:33,034-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:22:43,054-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 13:22:46,019-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 13:22:53,069-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 13:22:57,640-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 13:23:03,083-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:23:13,094-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:23:23,108-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 13:23:23,337-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 13:23:33,120-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 13:23:34,115-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 13:23:43,132-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:23:53,142-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 13:24:00,655-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 13:24:03,152-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:24:10,588-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 13:24:13,162-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:24:23,171-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:24:33,175-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:24:37,972-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 13:24:43,186-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 13:24:47,063-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 13:24:53,196-0600 INFO Multiplexer No streams
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-08 13:24:56,946-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:36330] for addition
-2013-11-08 13:24:56,946-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:36330] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:36330]
-2013-11-08 13:24:56,946-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:36330]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f2f
-2013-11-08 13:24:56,984-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f2f
-MetaChannel [context: cps-36330, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:36330]
-2013-11-08 13:24:56,984-0600 INFO MetaChannel MetaChannel [context: cps-36330, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:36330]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-08 13:24:56,984-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-08 13:24:57,024-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-08 13:24:57,024-0600 INFO Sender Sender TCPChannel queue size: 0
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-08 13:24:57,064-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:36331] for addition
-2013-11-08 13:24:57,065-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:36331] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:36331]
-2013-11-08 13:24:57,065-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:36331]
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f2e
-2013-11-08 13:24:57,085-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f2e
-MetaChannel [context: cps-36331, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:36331]
-2013-11-08 13:24:57,085-0600 INFO MetaChannel MetaChannel [context: cps-36331, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:36331]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-08 13:24:57,085-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-08 13:24:57,105-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-08 13:24:57,107-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 140.0) submitting urn:job-0-1383759275065-1383759275066: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-08 13:24:57,107-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 140.0) submitting urn:job-0-1383759275065-1383759275066: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-08 13:24:57,108-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-08 13:24:57,108-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-08 13:24:57,114-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7612, JOBSTATUS) unregistering (send)
-2013-11-08 13:24:57,114-0600 INFO RequestHandler Handler(tag: 7612, JOBSTATUS) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-08 13:24:57,125-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-08 13:24:57,166-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:36332] for addition
-2013-11-08 13:24:57,166-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:36332] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:36332]
-2013-11-08 13:24:57,166-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:36332]
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-08 13:24:57,206-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f2d
-2013-11-08 13:24:57,206-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f2d
-MetaChannel [context: cps-36332, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:36332]
-2013-11-08 13:24:57,206-0600 INFO MetaChannel MetaChannel [context: cps-36332, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:36332]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-08 13:24:57,207-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-08 13:24:57,209-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-crush.mcs.anl.gov:0 (quality: 54.0) submitting urn:job-1-1383759275068-1383759275069: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-08 13:24:57,209-0600 INFO Cpu crush.mcs.anl.gov:0 (quality: 54.0) submitting urn:job-1-1383759275068-1383759275069: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-08 13:24:57,216-0600 DEBUG Cpu Active workerid=crush.mcs.anl.gov:000000
-Handler(tag: 139, JOBSTATUS) unregistering (send)
-2013-11-08 13:24:57,216-0600 INFO RequestHandler Handler(tag: 139, JOBSTATUS) unregistering (send)
-Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-08 13:24:57,227-0600 INFO RequestHandler Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-08 13:24:59,171-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-08 13:24:59,171-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 13:24:59,171-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 13:24:59,171-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Channel config: null -> {keepalive=null}
-2013-11-08 13:24:59,171-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:36330] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 13:24:59,171-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:36330] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 7613, JOBSTATUS) unregistering (send)
-2013-11-08 13:24:59,171-0600 INFO RequestHandler Handler(tag: 7613, JOBSTATUS) unregistering (send)
-null: Channel shut down
-2013-11-08 13:24:59,172-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-36330, boundTo: TCPChannel [type: server, contact: 127.0.0.1:36330]] binding to NullChannel
-2013-11-08 13:24:59,172-0600 INFO MetaChannel MetaChannel [context: cps-36330, boundTo: TCPChannel [type: server, contact: 127.0.0.1:36330]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:36330]
-2013-11-08 13:24:59,172-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:36330]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 13:24:59,172-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-08 13:24:59,172-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 13:24:59,172-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 13:24:59,172-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-=> Command(tag: 2, JOBSTATUS)
-2013-11-08 13:24:59,172-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-2013-11-08 13:24:59,172-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 13:24:59,172-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-08 13:24:59,173-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Channel is closed. Removing.
-2013-11-08 13:24:59,173-0600 INFO Multiplexer Channel is closed. Removing.
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 13:24:59,173-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-08 13:24:59,173-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-08 13:24:59,173-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 13:24:59,173-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-08 13:24:59,173-0600 INFO ChannelManager Channel exception handled
-Channel config: null -> {keepalive=null}
-2013-11-08 13:24:59,173-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:36330] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-08 13:24:59,173-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:36330] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-08 13:24:59,173-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:36330]
-2013-11-08 13:24:59,173-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:36330]
-2013-11-08 13:24:59,359-0600 DEBUG Cpu Completed
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 13:24:59,359-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-crush.mcs.anl.gov:0 jobTerminated
-2013-11-08 13:24:59,359-0600 INFO Cpu crush.mcs.anl.gov:0 jobTerminated
-Channel config: null -> {keepalive=null}
-2013-11-08 13:24:59,359-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-2013-11-08 13:24:59,359-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-TCPChannel [type: server, contact: 127.0.0.1:36331] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 13:24:59,359-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:36331] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 140, JOBSTATUS) unregistering (send)
-2013-11-08 13:24:59,359-0600 INFO RequestHandler Handler(tag: 140, JOBSTATUS) unregistering (send)
-null: Channel shut down
-2013-11-08 13:24:59,359-0600 INFO AbstractTCPChannel null: Channel shut down
-2013-11-08 13:24:59,359-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 2
-MetaChannel [context: cps-36331, boundTo: TCPChannel [type: server, contact: 127.0.0.1:36331]] binding to NullChannel
-2013-11-08 13:24:59,360-0600 INFO MetaChannel MetaChannel [context: cps-36331, boundTo: TCPChannel [type: server, contact: 127.0.0.1:36331]] binding to NullChannel
-2013-11-08 13:24:59,360-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:36331]
-2013-11-08 13:24:59,360-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:36331]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 13:24:59,360-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-08 13:24:59,360-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 13:24:59,360-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-08 13:24:59,360-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 13:24:59,360-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-08 13:24:59,360-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 13:24:59,360-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-08 13:24:59,360-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 13:24:59,360-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-08 13:24:59,360-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-08 13:24:59,374-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-08 13:24:59,374-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-08 13:24:59,374-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:36331] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-08 13:24:59,374-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:36331] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-08 13:24:59,374-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:36331]
-2013-11-08 13:24:59,374-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:36331]
-Avg stream buf: 0
-2013-11-08 13:25:03,201-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-08 13:25:13,215-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 13:25:15,291-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-Avg stream buf: 0
-2013-11-08 13:25:23,228-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 13:25:23,537-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-Avg stream buf: 0
-2013-11-08 13:25:33,237-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-08 13:25:43,246-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 13:25:52,609-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-Avg stream buf: 0
-2013-11-08 13:25:53,255-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:26:00,010-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-08 13:26:03,264-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-08 13:26:13,272-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-08 13:26:23,281-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:26:29,926-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-08 13:26:33,290-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:26:36,484-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-08 13:26:43,306-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-08 13:26:53,318-0600 INFO Multiplexer Avg stream buf: 0
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-132457.105, now=131108-132657.946, channel=TCPChannel [type: server, contact: 127.0.0.1:36330]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 13:26:57,946-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-132457.105, now=131108-132657.946, channel=TCPChannel [type: server, contact: 127.0.0.1:36330]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-08 13:26:57,947-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-132457.105, now=131108-132657.946, channel=TCPChannel [type: server, contact: 127.0.0.1:36330]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 13:26:57,947-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-132457.105, now=131108-132657.946, channel=TCPChannel [type: server, contact: 127.0.0.1:36330]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-08 13:26:57,947-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-132457.105, now=131108-132657.946, channel=TCPChannel [type: server, contact: 127.0.0.1:36330]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 13:26:57,947-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-132457.105, now=131108-132657.946, channel=TCPChannel [type: server, contact: 127.0.0.1:36330]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-08 13:26:57,947-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-132457.105, now=131108-132657.946, channel=TCPChannel [type: server, contact: 127.0.0.1:36330]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 13:26:57,947-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-132457.105, now=131108-132657.946, channel=TCPChannel [type: server, contact: 127.0.0.1:36330]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-08 13:26:57,947-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-132457.105, now=131108-132657.946, channel=TCPChannel [type: server, contact: 127.0.0.1:36330]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 13:26:57,947-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-132457.105, now=131108-132657.946, channel=TCPChannel [type: server, contact: 127.0.0.1:36330]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-08 13:26:57,947-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:36330] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-132457.105, now=131108-132657.946, channel=TCPChannel [type: server, contact: 127.0.0.1:36330]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 13:26:57,947-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:36330] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-132457.105, now=131108-132657.946, channel=TCPChannel [type: server, contact: 127.0.0.1:36330]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-08 13:26:57,947-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-132457.206, now=131108-132658.065, channel=TCPChannel [type: server, contact: 127.0.0.1:36331]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 13:26:58,065-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-132457.206, now=131108-132658.065, channel=TCPChannel [type: server, contact: 127.0.0.1:36331]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-08 13:26:58,066-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-132457.206, now=131108-132658.065, channel=TCPChannel [type: server, contact: 127.0.0.1:36331]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 13:26:58,066-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-132457.206, now=131108-132658.065, channel=TCPChannel [type: server, contact: 127.0.0.1:36331]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-08 13:26:58,066-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-132457.206, now=131108-132658.065, channel=TCPChannel [type: server, contact: 127.0.0.1:36331]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 13:26:58,066-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-132457.206, now=131108-132658.065, channel=TCPChannel [type: server, contact: 127.0.0.1:36331]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-08 13:26:58,066-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-132457.206, now=131108-132658.065, channel=TCPChannel [type: server, contact: 127.0.0.1:36331]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 13:26:58,066-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-132457.206, now=131108-132658.065, channel=TCPChannel [type: server, contact: 127.0.0.1:36331]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-08 13:26:58,066-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-132457.206, now=131108-132658.065, channel=TCPChannel [type: server, contact: 127.0.0.1:36331]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 13:26:58,066-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-132457.206, now=131108-132658.065, channel=TCPChannel [type: server, contact: 127.0.0.1:36331]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-08 13:26:58,066-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:36331] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-132457.206, now=131108-132658.065, channel=TCPChannel [type: server, contact: 127.0.0.1:36331]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 13:26:58,066-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:36331] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-132457.206, now=131108-132658.065, channel=TCPChannel [type: server, contact: 127.0.0.1:36331]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-08 13:26:58,066-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-132457.227, now=131108-132658.166, channel=TCPChannel [type: server, contact: 127.0.0.1:36332]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 13:26:58,166-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-132457.227, now=131108-132658.166, channel=TCPChannel [type: server, contact: 127.0.0.1:36332]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-08 13:26:58,167-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:36332] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-132457.227, now=131108-132658.166, channel=TCPChannel [type: server, contact: 127.0.0.1:36332]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 13:26:58,167-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:36332] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-132457.227, now=131108-132658.166, channel=TCPChannel [type: server, contact: 127.0.0.1:36332]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-null: Channel shut down
-2013-11-08 13:26:58,167-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-36332, boundTo: TCPChannel [type: server, contact: 127.0.0.1:36332]] binding to NullChannel
-2013-11-08 13:26:58,167-0600 INFO MetaChannel MetaChannel [context: cps-36332, boundTo: TCPChannel [type: server, contact: 127.0.0.1:36332]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:36332]
-2013-11-08 13:26:58,167-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:36332]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-132457.227, now=131108-132658.166, channel=TCPChannel [type: server, contact: 127.0.0.1:36332]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 13:26:58,167-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-132457.227, now=131108-132658.166, channel=TCPChannel [type: server, contact: 127.0.0.1:36332]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel exception handled
-2013-11-08 13:26:58,167-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-08 13:26:58,173-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-08 13:26:58,173-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-08 13:26:58,173-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:36332] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-08 13:26:58,173-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:36332] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-08 13:26:58,173-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:36332]
-2013-11-08 13:26:58,173-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:36332]
-No streams
-2013-11-08 13:27:03,323-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 13:27:07,245-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:27:12,959-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 13:27:13,325-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:27:23,334-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:27:33,346-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:27:43,356-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 13:27:44,563-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 13:27:49,433-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 13:27:53,371-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:28:03,385-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:28:13,393-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 13:28:21,881-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 13:28:23,401-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:28:25,907-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 13:28:33,415-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:28:43,426-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:28:53,431-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:28:59,198-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 13:29:02,382-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 13:29:03,448-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:29:13,457-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:29:23,461-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:29:33,478-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 13:29:36,517-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 13:29:38,856-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 13:29:43,483-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:29:53,502-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:30:03,512-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:30:13,523-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:30:13,834-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 13:30:15,330-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 13:30:23,532-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:30:33,543-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:30:43,553-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 13:30:51,153-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 13:30:51,804-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 13:30:53,562-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:31:03,570-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:31:13,578-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:31:23,585-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:31:28,278-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:31:28,470-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 13:31:33,593-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:31:43,601-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:31:53,610-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:32:03,618-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 13:32:04,753-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 13:32:05,789-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 13:32:13,629-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:32:23,649-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:32:33,657-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:32:41,226-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 13:32:43,107-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 13:32:43,666-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:32:53,675-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:33:03,684-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:33:13,696-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:33:17,700-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:33:20,425-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 13:33:23,709-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:33:33,719-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:33:43,730-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:33:53,741-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:33:54,174-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:33:57,743-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 13:34:03,750-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:34:13,760-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:34:23,770-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:34:30,648-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 13:34:33,782-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:34:35,061-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 13:34:43,792-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:34:53,802-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:35:03,811-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 13:35:07,122-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:35:12,379-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 13:35:13,820-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:35:23,829-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:35:33,838-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:35:43,596-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 13:35:43,847-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:35:49,697-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 13:35:53,857-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:36:03,866-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:36:13,875-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:36:20,070-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 13:36:23,884-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 13:36:27,016-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 13:36:33,894-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:36:43,904-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:36:53,913-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 13:36:56,544-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 13:37:03,922-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:37:04,333-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 13:37:13,932-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:37:23,941-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 13:37:33,018-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 13:37:33,950-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 3ms
-2013-11-08 13:37:41,653-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 3ms
-No streams
-2013-11-08 13:37:43,960-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:37:53,970-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:38:03,978-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 13:38:09,493-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 13:38:13,991-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 13:38:18,970-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 13:38:24,005-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:38:34,016-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:38:44,026-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:38:45,966-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 13:38:54,037-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:38:56,287-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 13:39:04,045-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:39:14,046-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 13:39:22,441-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 13:39:24,060-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 13:39:33,606-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 13:39:34,062-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:39:44,077-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:39:54,089-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:39:58,915-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 13:40:04,100-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:40:10,923-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 13:40:14,108-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:40:24,115-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:40:34,123-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:40:35,389-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 13:40:44,131-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:40:48,241-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 13:40:54,139-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:41:04,147-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:41:11,863-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 13:41:14,154-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:41:24,162-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 13:41:25,560-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 13:41:34,170-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:41:44,178-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:41:48,337-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 13:41:54,185-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:42:02,877-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 13:42:04,193-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:42:14,200-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:42:24,208-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 13:42:24,812-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 13:42:34,216-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 3ms
-2013-11-08 13:42:40,197-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 3ms
-No streams
-2013-11-08 13:42:44,224-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:42:54,232-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:43:01,285-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 13:43:04,240-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:43:14,248-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:43:17,513-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 13:43:24,256-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:43:34,264-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:43:37,759-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 13:43:44,272-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:43:54,280-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:43:54,832-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 13:44:04,288-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:44:14,233-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 13:44:14,296-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:44:24,304-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 13:44:32,150-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 13:44:34,312-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:44:44,320-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 13:44:50,708-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 13:44:54,328-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:45:04,335-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:45:09,468-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 13:45:14,343-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:45:24,351-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:45:27,181-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 13:45:34,359-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:45:44,367-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:45:46,787-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 13:45:54,375-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:46:03,655-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 13:46:04,383-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:46:14,391-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:46:24,104-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 13:46:24,398-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:46:34,406-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 13:46:40,130-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 13:46:44,414-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:46:54,422-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:47:01,422-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 13:47:04,430-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:47:14,438-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 13:47:16,603-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 13:47:24,446-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:47:34,454-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 13:47:38,740-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 13:47:44,462-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 13:47:53,078-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 13:47:54,470-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:48:04,478-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:48:14,486-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:48:16,058-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 13:48:24,494-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 13:48:29,552-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 13:48:34,502-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:48:44,510-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:48:53,376-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 13:48:54,518-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:49:04,526-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:49:06,025-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 13:49:14,534-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:49:24,542-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 13:49:30,694-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 13:49:34,550-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 13:49:42,499-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 13:49:44,558-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:49:54,566-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:50:04,574-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:50:08,012-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 13:50:14,582-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 13:50:18,974-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 13:50:24,591-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:50:34,599-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:50:44,608-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 13:50:45,330-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 13:50:54,616-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 13:50:55,448-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 13:51:04,625-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:51:14,634-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:51:22,649-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 13:51:24,643-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 13:51:31,921-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 13:51:34,652-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:51:44,660-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:51:54,669-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:51:59,968-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 13:52:04,677-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 13:52:08,395-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 13:52:14,686-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:52:24,694-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:52:34,702-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 13:52:37,287-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 13:52:44,711-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:52:44,869-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 13:52:54,719-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:53:04,727-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:53:14,604-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 13:53:14,735-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:53:21,343-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 13:53:24,744-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:53:34,752-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:53:44,760-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 13:53:51,923-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 13:53:54,769-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 13:53:57,818-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 13:54:04,777-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:54:14,785-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:54:24,793-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:54:29,240-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:54:34,291-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 13:54:34,802-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:54:44,809-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:54:54,818-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:55:04,825-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:55:06,558-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:55:10,765-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 13:55:14,833-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:55:24,841-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:55:34,849-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 13:55:43,876-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 13:55:44,857-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 13:55:47,239-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 13:55:54,865-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:56:04,872-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:56:14,880-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 13:56:21,194-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:56:23,713-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 13:56:24,888-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:56:34,896-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:56:44,904-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:56:54,912-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:56:58,512-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:57:00,187-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 13:57:04,920-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:57:14,928-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:57:24,936-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:57:34,944-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 13:57:35,830-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 13:57:36,662-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 13:57:44,952-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:57:54,960-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:58:04,968-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 13:58:13,135-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:58:13,148-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 13:58:14,976-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:58:24,984-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:58:34,992-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:58:45,000-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:58:49,609-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 13:58:50,466-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 13:58:55,008-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:59:05,016-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:59:15,024-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:59:25,032-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 13:59:26,083-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 13:59:27,784-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 13:59:35,040-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:59:45,048-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 13:59:55,057-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:00:02,557-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:00:05,065-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:00:05,102-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:00:15,073-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:00:25,081-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:00:35,089-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:00:39,031-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:00:42,421-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:00:45,098-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:00:55,106-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:01:05,114-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:01:15,122-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 14:01:19,740-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 4766ms
-2013-11-08 14:01:20,271-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 4766ms
-No streams
-2013-11-08 14:01:25,131-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:01:35,139-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:01:45,147-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 14:01:51,979-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 14:01:55,155-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:01:57,057-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:02:05,163-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:02:15,172-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:02:25,180-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 572ms
-2013-11-08 14:02:29,025-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 572ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 14:02:34,376-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 14:02:35,188-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:02:45,196-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:02:55,204-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:03:04,928-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:03:05,212-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:03:11,693-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:03:15,220-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:03:25,228-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:03:35,236-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 14:03:41,403-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 14:03:45,244-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 589ms
-2013-11-08 14:03:49,599-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 589ms
-No streams
-2013-11-08 14:03:55,259-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:04:05,267-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:04:15,275-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 14:04:17,876-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 14:04:25,283-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 14:04:26,329-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 14:04:35,291-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:04:45,303-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 14:04:54,351-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 14:04:55,311-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:05:03,647-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:05:05,319-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:05:15,327-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:05:25,335-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:05:30,824-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:05:35,348-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 14:05:40,965-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 14:05:45,356-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:05:55,364-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:06:05,372-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 14:06:07,299-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 14:06:15,380-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 14:06:18,283-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 14:06:25,396-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:06:35,404-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:06:43,773-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:06:45,412-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:06:55,420-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1330ms
-2013-11-08 14:06:56,930-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1330ms
-No streams
-2013-11-08 14:07:05,434-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:07:15,442-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 14:07:20,247-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 14:07:25,450-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:07:32,919-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:07:35,458-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:07:45,466-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:07:55,474-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:07:56,721-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:08:05,482-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:08:10,237-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:08:15,491-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:08:25,499-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 14:08:33,195-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 14:08:35,506-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:08:45,526-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 985ms
-2013-11-08 14:08:48,539-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 985ms
-No streams
-2013-11-08 14:08:55,534-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:09:05,542-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:09:09,669-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:09:15,550-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 14:09:24,873-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 14:09:25,558-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:09:35,566-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:09:45,574-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:09:46,143-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:09:55,582-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:10:02,191-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:10:05,590-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:10:15,598-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 14:10:22,618-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 14:10:25,606-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:10:35,615-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 14:10:39,510-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 14:10:45,634-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:10:55,642-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 14:10:59,091-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 14:11:05,650-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:11:15,658-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 14:11:16,827-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 14:11:25,666-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 14:11:35,566-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 14:11:35,680-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:11:45,688-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:11:54,145-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:11:55,696-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:12:05,704-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:12:12,039-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:12:15,712-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:12:25,720-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 14:12:31,464-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 14:12:35,728-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:12:45,737-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:12:48,513-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:12:55,746-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:13:05,754-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:13:08,781-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:13:15,763-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:13:24,987-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:13:25,771-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:13:35,778-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:13:45,787-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 3ms
-2013-11-08 14:13:46,101-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 3ms
-No streams
-2013-11-08 14:13:55,795-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 14:14:01,462-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 14:14:05,801-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:14:15,809-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 14:14:23,417-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 14:14:25,815-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:14:35,823-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 213ms
-2013-11-08 14:14:38,147-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 213ms
-No streams
-2013-11-08 14:14:45,828-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:14:55,836-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:15:00,735-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:15:05,843-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 14:15:14,410-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 14:15:15,852-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:15:25,859-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:15:35,867-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 14:15:38,054-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 14:15:45,884-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 14:15:50,884-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 14:15:55,893-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:16:05,901-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 14:16:15,372-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 14:16:15,909-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:16:25,916-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:16:27,357-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:16:35,924-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:16:45,932-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 14:16:52,689-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 14:16:55,940-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:17:03,831-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:17:05,948-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:17:15,956-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:17:25,964-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 14:17:30,007-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 14:17:35,972-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:17:40,305-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:17:45,980-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:17:55,988-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:18:05,996-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 14:18:07,326-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 14:18:16,004-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 14:18:16,780-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 14:18:26,012-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:18:36,020-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 597ms
-2013-11-08 14:18:45,239-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 597ms
-No streams
-2013-11-08 14:18:46,028-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 14:18:53,254-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 14:18:56,036-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:19:06,055-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:19:16,063-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:19:21,962-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:19:26,071-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 14:19:29,727-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 14:19:36,078-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:19:46,086-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:19:56,094-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 14:19:59,280-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 14:20:06,099-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 14:20:06,202-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 14:20:16,107-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:20:26,115-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:20:36,123-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 14:20:36,598-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 14:20:42,675-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 14:20:46,132-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:20:56,140-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:21:06,149-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:21:13,916-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:21:16,157-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 14:21:19,150-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 14:21:26,165-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:21:36,184-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:21:46,192-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 14:21:51,235-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:21:55,623-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:21:56,200-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:22:06,208-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:22:16,216-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:22:26,224-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 14:22:28,553-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-08 14:22:30,121-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:38652] for addition
-2013-11-08 14:22:30,122-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:38652] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:38652]
-2013-11-08 14:22:30,122-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:38652]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f2c
-2013-11-08 14:22:30,153-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f2c
-MetaChannel [context: cps-38652, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:38652]
-2013-11-08 14:22:30,153-0600 INFO MetaChannel MetaChannel [context: cps-38652, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:38652]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-08 14:22:30,153-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-08 14:22:30,194-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-08 14:22:30,194-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-08 14:22:30,232-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:38653] for addition
-2013-11-08 14:22:30,232-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:38653] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:38653]
-2013-11-08 14:22:30,232-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:38653]
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f2b
-2013-11-08 14:22:30,234-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f2b
-MetaChannel [context: cps-38653, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:38653]
-2013-11-08 14:22:30,234-0600 INFO MetaChannel MetaChannel [context: cps-38653, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:38653]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-08 14:22:30,234-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-08 14:22:30,275-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-08 14:22:30,275-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-08 14:22:30,299-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 141.0) submitting urn:job-0-1383759275071-1383759275072: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-08 14:22:30,299-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 141.0) submitting urn:job-0-1383759275071-1383759275072: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-08 14:22:30,300-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-08 14:22:30,300-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-08 14:22:30,306-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7614, JOBSTATUS) unregistering (send)
-2013-11-08 14:22:30,306-0600 INFO RequestHandler Handler(tag: 7614, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-08 14:22:30,316-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:38654] for addition
-2013-11-08 14:22:30,316-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:38654] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:38654]
-2013-11-08 14:22:30,316-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:38654]
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-08 14:22:30,356-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f2a
-2013-11-08 14:22:30,356-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f2a
-MetaChannel [context: cps-38654, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:38654]
-2013-11-08 14:22:30,356-0600 INFO MetaChannel MetaChannel [context: cps-38654, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:38654]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-08 14:22:30,356-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-08 14:22:30,377-0600 INFO RequestHandler Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-08 14:22:30,400-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-crush.mcs.anl.gov:0 (quality: 55.0) submitting urn:job-1-1383759275074-1383759275075: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-08 14:22:30,400-0600 INFO Cpu crush.mcs.anl.gov:0 (quality: 55.0) submitting urn:job-1-1383759275074-1383759275075: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-08 14:22:30,407-0600 DEBUG Cpu Active workerid=crush.mcs.anl.gov:000000
-Handler(tag: 141, JOBSTATUS) unregistering (send)
-2013-11-08 14:22:30,407-0600 INFO RequestHandler Handler(tag: 141, JOBSTATUS) unregistering (send)
-Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-08 14:22:30,457-0600 INFO RequestHandler Handler(tag: 42566, SUBMITJOB) unregistering (send)
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:22:32,097-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:22:32,532-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-08 14:22:32,532-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-08 14:22:32,532-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Handler(tag: 7615, JOBSTATUS) unregistering (send)
-2013-11-08 14:22:32,532-0600 INFO RequestHandler Handler(tag: 7615, JOBSTATUS) unregistering (send)
-2013-11-08 14:22:32,532-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 142.0) submitting urn:job-2-1383759275077-1383759275078: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-08 14:22:32,532-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 142.0) submitting urn:job-2-1383759275077-1383759275078: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-08 14:22:32,537-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7616, JOBSTATUS) unregistering (send)
-2013-11-08 14:22:32,538-0600 INFO RequestHandler Handler(tag: 7616, JOBSTATUS) unregistering (send)
-2013-11-08 14:22:32,595-0600 DEBUG Cpu Completed
-crush.mcs.anl.gov:0 jobTerminated
-2013-11-08 14:22:32,595-0600 INFO Cpu crush.mcs.anl.gov:0 jobTerminated
-2013-11-08 14:22:32,595-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-Handler(tag: 142, JOBSTATUS) unregistering (send)
-2013-11-08 14:22:32,595-0600 INFO RequestHandler Handler(tag: 142, JOBSTATUS) unregistering (send)
-2013-11-08 14:22:32,595-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-08 14:22:32,595-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-Avg stream buf: 0
-2013-11-08 14:22:36,237-0600 INFO Multiplexer Avg stream buf: 0
-2013-11-08 14:22:38,019-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-08 14:22:38,019-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-08 14:22:38,019-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Handler(tag: 7617, JOBSTATUS) unregistering (send)
-2013-11-08 14:22:38,019-0600 INFO RequestHandler Handler(tag: 7617, JOBSTATUS) unregistering (send)
-2013-11-08 14:22:38,019-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-2013-11-08 14:22:38,019-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-Avg stream buf: 0
-2013-11-08 14:22:46,247-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-08 14:22:56,258-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 14:23:05,870-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-08 14:23:06,267-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 14:23:08,571-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-08 14:23:16,275-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-08 14:23:26,284-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-08 14:23:36,294-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 14:23:43,189-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 14:23:45,045-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-Avg stream buf: 0
-2013-11-08 14:23:46,303-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-08 14:23:56,313-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-08 14:24:06,322-0600 INFO Multiplexer Avg stream buf: 0
-Avg stream buf: 0
-2013-11-08 14:24:16,331-0600 INFO Multiplexer Avg stream buf: 0
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 14:24:20,506-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 14:24:21,520-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-Avg stream buf: 0
-2013-11-08 14:24:26,340-0600 INFO Multiplexer Avg stream buf: 0
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.275, now=131108-142431.122, channel=TCPChannel [type: server, contact: 127.0.0.1:38652]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 14:24:31,122-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.275, now=131108-142431.122, channel=TCPChannel [type: server, contact: 127.0.0.1:38652]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-08 14:24:31,123-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.275, now=131108-142431.122, channel=TCPChannel [type: server, contact: 127.0.0.1:38652]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 14:24:31,123-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.275, now=131108-142431.122, channel=TCPChannel [type: server, contact: 127.0.0.1:38652]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-08 14:24:31,123-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.275, now=131108-142431.122, channel=TCPChannel [type: server, contact: 127.0.0.1:38652]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 14:24:31,123-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.275, now=131108-142431.122, channel=TCPChannel [type: server, contact: 127.0.0.1:38652]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-08 14:24:31,123-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.275, now=131108-142431.122, channel=TCPChannel [type: server, contact: 127.0.0.1:38652]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 14:24:31,123-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.275, now=131108-142431.122, channel=TCPChannel [type: server, contact: 127.0.0.1:38652]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-08 14:24:31,123-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.275, now=131108-142431.122, channel=TCPChannel [type: server, contact: 127.0.0.1:38652]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 14:24:31,123-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.275, now=131108-142431.122, channel=TCPChannel [type: server, contact: 127.0.0.1:38652]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-08 14:24:31,123-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:38652] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.275, now=131108-142431.122, channel=TCPChannel [type: server, contact: 127.0.0.1:38652]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 14:24:31,124-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:38652] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.275, now=131108-142431.122, channel=TCPChannel [type: server, contact: 127.0.0.1:38652]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-null: Channel shut down
-2013-11-08 14:24:31,124-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-38652, boundTo: TCPChannel [type: server, contact: 127.0.0.1:38652]] binding to NullChannel
-2013-11-08 14:24:31,124-0600 INFO MetaChannel MetaChannel [context: cps-38652, boundTo: TCPChannel [type: server, contact: 127.0.0.1:38652]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:38652]
-2013-11-08 14:24:31,124-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:38652]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.275, now=131108-142431.122, channel=TCPChannel [type: server, contact: 127.0.0.1:38652]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 14:24:31,124-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.275, now=131108-142431.122, channel=TCPChannel [type: server, contact: 127.0.0.1:38652]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-08 14:24:31,124-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.275, now=131108-142431.122, channel=TCPChannel [type: server, contact: 127.0.0.1:38652]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 14:24:31,124-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.275, now=131108-142431.122, channel=TCPChannel [type: server, contact: 127.0.0.1:38652]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-08 14:24:31,124-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.275, now=131108-142431.122, channel=TCPChannel [type: server, contact: 127.0.0.1:38652]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 14:24:31,124-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.275, now=131108-142431.122, channel=TCPChannel [type: server, contact: 127.0.0.1:38652]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-08 14:24:31,124-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.275, now=131108-142431.122, channel=TCPChannel [type: server, contact: 127.0.0.1:38652]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 14:24:31,124-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.275, now=131108-142431.122, channel=TCPChannel [type: server, contact: 127.0.0.1:38652]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-08 14:24:31,124-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.275, now=131108-142431.122, channel=TCPChannel [type: server, contact: 127.0.0.1:38652]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 14:24:31,124-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.275, now=131108-142431.122, channel=TCPChannel [type: server, contact: 127.0.0.1:38652]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel exception handled
-2013-11-08 14:24:31,124-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-08 14:24:31,134-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-08 14:24:31,134-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-08 14:24:31,134-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:38652] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-08 14:24:31,134-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:38652] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-08 14:24:31,134-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:38652]
-2013-11-08 14:24:31,134-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:38652]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.356, now=131108-142431.232, channel=TCPChannel [type: server, contact: 127.0.0.1:38653]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 14:24:31,233-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.356, now=131108-142431.232, channel=TCPChannel [type: server, contact: 127.0.0.1:38653]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-08 14:24:31,233-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.356, now=131108-142431.232, channel=TCPChannel [type: server, contact: 127.0.0.1:38653]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 14:24:31,233-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.356, now=131108-142431.232, channel=TCPChannel [type: server, contact: 127.0.0.1:38653]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-08 14:24:31,233-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.356, now=131108-142431.232, channel=TCPChannel [type: server, contact: 127.0.0.1:38653]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 14:24:31,233-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.356, now=131108-142431.232, channel=TCPChannel [type: server, contact: 127.0.0.1:38653]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-08 14:24:31,233-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.356, now=131108-142431.232, channel=TCPChannel [type: server, contact: 127.0.0.1:38653]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 14:24:31,233-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.356, now=131108-142431.232, channel=TCPChannel [type: server, contact: 127.0.0.1:38653]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-08 14:24:31,233-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.356, now=131108-142431.232, channel=TCPChannel [type: server, contact: 127.0.0.1:38653]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 14:24:31,233-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.356, now=131108-142431.232, channel=TCPChannel [type: server, contact: 127.0.0.1:38653]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {}
-2013-11-08 14:24:31,233-0600 INFO AbstractCoasterChannel Channel config: null -> {}
-TCPChannel [type: server, contact: 127.0.0.1:38653] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.356, now=131108-142431.232, channel=TCPChannel [type: server, contact: 127.0.0.1:38653]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 14:24:31,233-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:38653] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.356, now=131108-142431.232, channel=TCPChannel [type: server, contact: 127.0.0.1:38653]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-null: Channel shut down
-2013-11-08 14:24:31,233-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-38653, boundTo: TCPChannel [type: server, contact: 127.0.0.1:38653]] binding to NullChannel
-2013-11-08 14:24:31,233-0600 INFO MetaChannel MetaChannel [context: cps-38653, boundTo: TCPChannel [type: server, contact: 127.0.0.1:38653]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:38653]
-2013-11-08 14:24:31,233-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:38653]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.356, now=131108-142431.232, channel=TCPChannel [type: server, contact: 127.0.0.1:38653]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 14:24:31,233-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.356, now=131108-142431.232, channel=TCPChannel [type: server, contact: 127.0.0.1:38653]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-08 14:24:31,233-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.356, now=131108-142431.232, channel=TCPChannel [type: server, contact: 127.0.0.1:38653]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 14:24:31,233-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.356, now=131108-142431.232, channel=TCPChannel [type: server, contact: 127.0.0.1:38653]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-08 14:24:31,233-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.356, now=131108-142431.232, channel=TCPChannel [type: server, contact: 127.0.0.1:38653]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 14:24:31,234-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.356, now=131108-142431.232, channel=TCPChannel [type: server, contact: 127.0.0.1:38653]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-08 14:24:31,234-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.356, now=131108-142431.232, channel=TCPChannel [type: server, contact: 127.0.0.1:38653]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 14:24:31,234-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.356, now=131108-142431.232, channel=TCPChannel [type: server, contact: 127.0.0.1:38653]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-08 14:24:31,234-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.356, now=131108-142431.232, channel=TCPChannel [type: server, contact: 127.0.0.1:38653]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 14:24:31,234-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.356, now=131108-142431.232, channel=TCPChannel [type: server, contact: 127.0.0.1:38653]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel exception handled
-2013-11-08 14:24:31,234-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-08 14:24:31,234-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-08 14:24:31,235-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {}
-2013-11-08 14:24:31,235-0600 INFO AbstractCoasterChannel Channel config: null -> {}
-TCPChannel [type: server, contact: 127.0.0.1:38653] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-08 14:24:31,235-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:38653] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-08 14:24:31,235-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:38653]
-2013-11-08 14:24:31,235-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:38653]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.457, now=131108-142431.316, channel=TCPChannel [type: server, contact: 127.0.0.1:38654]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 14:24:31,316-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.457, now=131108-142431.316, channel=TCPChannel [type: server, contact: 127.0.0.1:38654]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-08 14:24:31,316-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.457, now=131108-142431.316, channel=TCPChannel [type: server, contact: 127.0.0.1:38654]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 14:24:31,316-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.457, now=131108-142431.316, channel=TCPChannel [type: server, contact: 127.0.0.1:38654]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-08 14:24:31,316-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.457, now=131108-142431.316, channel=TCPChannel [type: server, contact: 127.0.0.1:38654]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 14:24:31,316-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.457, now=131108-142431.316, channel=TCPChannel [type: server, contact: 127.0.0.1:38654]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-08 14:24:31,316-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.457, now=131108-142431.316, channel=TCPChannel [type: server, contact: 127.0.0.1:38654]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 14:24:31,316-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.457, now=131108-142431.316, channel=TCPChannel [type: server, contact: 127.0.0.1:38654]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-08 14:24:31,316-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.457, now=131108-142431.316, channel=TCPChannel [type: server, contact: 127.0.0.1:38654]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 14:24:31,316-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.457, now=131108-142431.316, channel=TCPChannel [type: server, contact: 127.0.0.1:38654]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-08 14:24:31,316-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:38654] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.457, now=131108-142431.316, channel=TCPChannel [type: server, contact: 127.0.0.1:38654]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 14:24:31,316-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:38654] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.457, now=131108-142431.316, channel=TCPChannel [type: server, contact: 127.0.0.1:38654]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-null: Channel shut down
-2013-11-08 14:24:31,317-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-38654, boundTo: TCPChannel [type: server, contact: 127.0.0.1:38654]] binding to NullChannel
-2013-11-08 14:24:31,317-0600 INFO MetaChannel MetaChannel [context: cps-38654, boundTo: TCPChannel [type: server, contact: 127.0.0.1:38654]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:38654]
-2013-11-08 14:24:31,317-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:38654]
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.457, now=131108-142431.316, channel=TCPChannel [type: server, contact: 127.0.0.1:38654]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 14:24:31,317-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.457, now=131108-142431.316, channel=TCPChannel [type: server, contact: 127.0.0.1:38654]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-08 14:24:31,317-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.457, now=131108-142431.316, channel=TCPChannel [type: server, contact: 127.0.0.1:38654]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 14:24:31,317-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.457, now=131108-142431.316, channel=TCPChannel [type: server, contact: 127.0.0.1:38654]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-08 14:24:31,317-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.457, now=131108-142431.316, channel=TCPChannel [type: server, contact: 127.0.0.1:38654]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 14:24:31,317-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.457, now=131108-142431.316, channel=TCPChannel [type: server, contact: 127.0.0.1:38654]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-08 14:24:31,317-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.457, now=131108-142431.316, channel=TCPChannel [type: server, contact: 127.0.0.1:38654]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 14:24:31,317-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.457, now=131108-142431.316, channel=TCPChannel [type: server, contact: 127.0.0.1:38654]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-08 14:24:31,317-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.457, now=131108-142431.316, channel=TCPChannel [type: server, contact: 127.0.0.1:38654]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 14:24:31,317-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-142230.457, now=131108-142431.316, channel=TCPChannel [type: server, contact: 127.0.0.1:38654]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel exception handled
-2013-11-08 14:24:31,317-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-08 14:24:31,335-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-08 14:24:31,335-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-08 14:24:31,335-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:38654] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-08 14:24:31,335-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:38654] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-08 14:24:31,335-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:38654]
-2013-11-08 14:24:31,335-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:38654]
-No streams
-2013-11-08 14:24:36,349-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:24:46,357-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:24:56,366-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 14:24:57,825-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:24:57,993-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:25:06,374-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:25:16,382-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:25:26,390-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:25:34,467-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 14:25:35,142-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 14:25:36,398-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:25:46,406-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:25:56,415-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:26:06,423-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:26:10,941-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 14:26:12,460-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 14:26:16,431-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:26:26,439-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:26:36,447-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:26:46,455-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:26:47,415-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 14:26:49,779-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 14:26:56,463-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:27:06,471-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:27:16,479-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 14:27:23,890-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 14:27:26,487-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 14:27:27,096-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 14:27:36,495-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:27:46,503-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:27:56,512-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:28:00,363-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 14:28:04,414-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 14:28:06,520-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:28:16,528-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:28:26,536-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:28:36,544-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 14:28:36,838-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 14:28:41,732-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 14:28:46,553-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:28:56,561-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:29:06,569-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:29:13,311-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:29:16,577-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:29:19,050-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:29:26,585-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:29:36,593-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:29:46,601-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 14:29:49,785-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:29:56,368-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:29:56,610-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:30:06,618-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:30:16,626-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 14:30:26,260-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 14:30:26,634-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 14:30:33,686-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 14:30:36,642-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:30:46,650-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:30:56,659-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 14:31:02,733-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 14:31:06,667-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 14:31:11,004-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 14:31:16,675-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:31:26,683-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:31:36,691-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 14:31:39,208-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 14:31:46,700-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:31:48,323-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:31:56,708-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:32:06,717-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 14:32:15,681-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 14:32:16,725-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 14:32:25,641-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 14:32:26,734-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:32:36,742-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:32:46,751-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:32:52,155-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:32:56,759-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 14:33:02,959-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 14:33:06,768-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:33:16,776-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:33:26,784-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 14:33:28,629-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 14:33:36,793-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 14:33:40,277-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 14:33:46,802-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:33:56,810-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:34:05,103-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:34:06,818-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:34:16,826-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 14:34:17,595-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 14:34:26,835-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:34:36,843-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:34:41,577-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:34:46,851-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:34:54,914-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:34:56,858-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:35:06,866-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:35:16,874-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 14:35:18,052-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 14:35:26,883-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 14:35:32,233-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 14:35:36,891-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:35:46,899-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:35:54,525-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:35:56,907-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:36:06,915-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 14:36:09,550-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 14:36:16,923-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:36:26,931-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:36:30,999-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:36:36,940-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 14:36:46,869-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 14:36:46,947-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:36:56,955-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:37:06,964-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:37:07,473-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:37:16,972-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 14:37:24,186-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 14:37:26,981-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:37:36,989-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 14:37:43,947-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 14:37:46,997-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:37:57,005-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 14:38:01,504-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 14:38:07,013-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:38:17,022-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:38:20,421-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:38:27,030-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:38:37,038-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 14:38:38,822-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 14:38:47,046-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 14:38:56,894-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 14:38:57,054-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:39:07,062-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 14:39:16,140-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 14:39:17,070-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:39:27,078-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:39:33,369-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:39:37,087-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:39:47,095-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 14:39:53,458-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 14:39:57,104-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:40:07,112-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 14:40:09,842-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 14:40:17,120-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:40:27,128-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 14:40:30,777-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 14:40:37,136-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:40:46,317-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:40:47,144-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:40:57,152-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:41:07,160-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 14:41:08,094-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 14:41:17,168-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:41:22,791-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:41:27,176-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:41:37,184-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 14:41:45,413-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 14:41:47,192-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:41:57,200-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:41:59,265-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:42:07,209-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:42:17,217-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 14:42:22,730-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 14:42:27,225-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 14:42:35,738-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 14:42:37,233-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:42:47,241-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:42:57,250-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 14:43:00,048-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 14:43:07,258-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:43:12,213-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:43:17,266-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:43:27,274-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:43:37,283-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 14:43:37,367-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 14:43:47,291-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:43:48,687-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:43:57,299-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:44:07,307-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 14:44:14,684-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 14:44:17,315-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:44:25,161-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:44:27,323-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:44:37,331-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:44:47,339-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:44:52,003-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:44:57,347-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:45:01,635-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:45:07,355-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:45:17,363-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:45:27,371-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 14:45:29,320-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 14:45:37,379-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-2013-11-08 14:45:38,110-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 2ms
-No streams
-2013-11-08 14:45:47,387-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:45:57,394-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 14:46:06,638-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 14:46:07,402-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:46:14,583-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:46:17,410-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:46:27,418-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:46:37,426-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-08 14:46:43,957-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-No streams
-2013-11-08 14:46:47,434-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:46:51,057-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:46:57,442-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:47:07,450-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:47:17,458-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 14:47:21,274-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 14:47:27,466-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:47:27,531-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:47:37,474-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:47:47,482-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:47:57,490-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 14:47:58,592-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:48:04,005-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:48:07,498-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:48:17,506-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:48:27,514-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 14:48:35,911-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 14:48:37,522-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:48:40,479-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:48:47,530-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:48:57,538-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:49:07,546-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 14:49:13,228-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:49:16,953-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:49:17,555-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:49:27,563-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:49:37,571-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:49:47,579-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:49:50,547-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:49:53,427-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:49:57,587-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:50:07,595-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:50:17,603-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:50:27,611-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 14:50:27,866-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-2013-11-08 14:50:29,901-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 1ms, rtt: 1ms
-No streams
-2013-11-08 14:50:37,619-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:50:47,627-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:50:57,635-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 14:51:05,183-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:51:06,376-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:51:07,644-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:51:17,651-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:51:27,660-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:51:37,668-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:51:42,502-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:51:42,849-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:51:47,676-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:51:57,684-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:52:07,692-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:52:17,700-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:52:19,323-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 14:52:19,820-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 14:52:27,708-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:52:37,716-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:52:47,724-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 14:52:55,798-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 14:52:57,138-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 14:52:57,732-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:53:07,740-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:53:17,748-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:53:27,756-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 14:53:32,273-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 14:53:34,456-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 14:53:37,764-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:53:47,772-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:53:57,781-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:54:07,789-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:54:08,746-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 14:54:11,775-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 14:54:17,797-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:54:27,805-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:54:37,813-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:54:45,220-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:54:47,821-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 14:54:49,093-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 14:54:57,829-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:55:07,837-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:55:17,845-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 14:55:21,695-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 14:55:26,412-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 14:55:27,853-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:55:37,861-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:55:47,869-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:55:57,877-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:55:58,168-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 14:56:03,729-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 14:56:07,885-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:56:17,893-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:56:27,901-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:56:34,642-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:56:37,909-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 14:56:41,047-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 14:56:47,917-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:56:57,925-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:57:07,934-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-2013-11-08 14:57:11,116-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 0ms
-No streams
-2013-11-08 14:57:17,942-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 14:57:18,366-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 14:57:27,949-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:57:37,957-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:57:47,591-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:57:47,965-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-2013-11-08 14:57:55,684-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-No streams
-2013-11-08 14:57:57,973-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:58:07,981-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:58:17,990-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:58:24,065-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:58:27,998-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 14:58:33,001-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 14:58:38,006-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:58:48,013-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:58:58,021-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 14:59:00,540-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 14:59:08,029-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 14:59:10,319-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 14:59:18,037-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:59:28,045-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 14:59:37,013-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 14:59:38,053-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 14:59:47,638-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 14:59:48,060-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 14:59:58,068-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:00:08,076-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 15:00:13,488-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 15:00:18,084-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:00:24,955-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:00:28,092-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:00:38,100-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:00:48,108-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:00:49,962-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:00:58,115-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:01:02,273-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:01:08,124-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:01:18,132-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 15:01:26,435-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 15:01:28,140-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:01:38,148-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 15:01:39,592-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 15:01:48,156-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:01:58,164-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 15:02:02,909-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 15:02:08,172-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:02:16,909-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:02:18,180-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:02:28,188-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:02:38,196-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 15:02:39,383-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 15:02:48,204-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:02:54,227-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:02:58,213-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:03:08,221-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 15:03:15,857-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 15:03:18,229-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:03:28,237-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:03:31,545-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:03:38,245-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:03:48,253-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:03:52,331-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:03:58,261-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:04:08,269-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 15:04:08,864-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 15:04:18,277-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:04:28,285-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 15:04:28,805-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 15:04:38,293-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 15:04:46,182-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 15:04:48,302-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:04:58,310-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 15:05:05,279-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 15:05:08,318-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:05:18,326-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 15:05:23,500-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 15:05:28,334-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:05:38,342-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:05:41,753-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:05:48,350-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:05:58,358-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:06:00,817-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:06:08,366-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 15:06:18,228-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 15:06:18,374-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:06:28,382-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:06:38,135-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:06:38,390-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:06:48,398-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 15:06:54,701-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 15:06:58,406-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:07:08,414-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-2013-11-08 15:07:15,454-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-No streams
-2013-11-08 15:07:18,422-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:07:28,429-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 15:07:31,175-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 15:07:38,437-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:07:48,445-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:07:52,771-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:07:58,453-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 15:08:07,649-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 15:08:08,461-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:08:18,469-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:08:28,477-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:08:30,089-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:08:38,485-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 15:08:44,123-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 15:08:48,493-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:08:58,501-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:09:07,408-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:09:08,509-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:09:18,517-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 15:09:20,597-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 15:09:28,525-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:09:38,533-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-2013-11-08 15:09:44,726-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-No streams
-2013-11-08 15:09:48,540-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 15:09:57,071-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 15:09:58,548-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:10:08,556-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:10:18,564-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:10:22,043-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:10:28,572-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 49ms
-2013-11-08 15:10:33,593-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 49ms
-No streams
-2013-11-08 15:10:38,580-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:10:48,588-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:10:58,596-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:10:59,361-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:11:08,604-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:11:10,019-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:11:18,612-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:11:28,620-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-2013-11-08 15:11:36,680-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-No streams
-2013-11-08 15:11:38,628-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 15:11:46,493-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 15:11:48,635-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:11:58,643-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:12:08,651-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:12:13,997-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:12:18,659-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 15:12:22,968-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 15:12:28,667-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:12:38,675-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:12:48,684-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-2013-11-08 15:12:51,316-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-No streams
-2013-11-08 15:12:58,692-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 15:12:59,443-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 15:13:08,700-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:13:18,708-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:13:28,634-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:13:28,716-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 15:13:35,916-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 15:13:38,724-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:13:48,732-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:13:58,740-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:14:05,951-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:14:08,748-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 15:14:12,391-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 15:14:18,756-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:14:28,765-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:14:38,773-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:14:43,269-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:14:48,781-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:14:48,865-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:14:58,789-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:15:08,797-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:15:18,805-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:15:20,587-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 15:15:25,339-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 15:15:28,813-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:15:38,821-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:15:48,829-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:15:57,905-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:15:58,837-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 15:16:01,813-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 15:16:08,845-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:16:18,853-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:16:28,861-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-2013-11-08 15:16:35,224-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 15:16:38,287-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 15:16:38,869-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:16:48,877-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:16:58,885-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:17:08,894-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:17:12,541-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 15:17:14,762-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 15:17:18,902-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:17:28,910-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:17:38,918-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:17:48,927-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:17:49,860-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 15:17:51,235-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 15:17:58,935-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:18:08,944-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:18:18,952-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-2013-11-08 15:18:27,178-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:18:27,709-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:18:28,960-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:18:38,969-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:18:48,977-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:18:58,985-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:19:04,183-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:19:04,495-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:19:08,993-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:19:19,002-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:19:29,010-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:19:39,018-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:19:40,657-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:19:41,813-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:19:49,026-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:19:59,035-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:20:09,043-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 15:20:17,131-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 15:20:19,051-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:20:19,131-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:20:29,059-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:20:39,067-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:20:49,075-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 15:20:53,605-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:20:56,449-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:20:59,084-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:21:09,092-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:21:19,100-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:21:29,108-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:21:30,079-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:21:33,767-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:21:39,116-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:21:49,125-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:21:59,133-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:22:06,553-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:22:09,142-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:22:11,085-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-08 15:22:13,706-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:38666] for addition
-2013-11-08 15:22:13,707-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:38666] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:38666]
-2013-11-08 15:22:13,707-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:38666]
-Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f29
-2013-11-08 15:22:13,736-0600 INFO ChannelConfigurationHandler Channel id: channel-1-u29cf5f3a-1422e7ae15d--7f29
-MetaChannel [context: cps-38666, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:38666]
-2013-11-08 15:22:13,736-0600 INFO MetaChannel MetaChannel [context: cps-38666, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:38666]
-Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-2013-11-08 15:22:13,736-0600 INFO RequestHandler Handler(tag: 38906, CHANNELCONFIG) unregistering (send)
-Sender TCPChannel queue size: 0
-2013-11-08 15:22:13,777-0600 INFO Sender Sender TCPChannel queue size: 0
-Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-2013-11-08 15:22:13,777-0600 INFO RequestHandler Handler(tag: 38907, CONFIGSERVICE) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-08 15:22:13,816-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:38667] for addition
-2013-11-08 15:22:13,816-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:38667] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:38667]
-2013-11-08 15:22:13,816-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:38667]
-Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f28
-2013-11-08 15:22:13,817-0600 INFO ChannelConfigurationHandler Channel id: channel-2-u29cf5f3a-1422e7ae15d--7f28
-MetaChannel [context: cps-38667, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:38667]
-2013-11-08 15:22:13,817-0600 INFO MetaChannel MetaChannel [context: cps-38667, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:38667]
-Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-2013-11-08 15:22:13,817-0600 INFO RequestHandler Handler(tag: 31103, CHANNELCONFIG) unregistering (send)
-Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-2013-11-08 15:22:13,858-0600 INFO RequestHandler Handler(tag: 31104, CONFIGSERVICE) unregistering (send)
-Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-08 15:22:13,858-0600 INFO RequestHandler Handler(tag: 38908, SUBMITJOB) unregistering (send)
-2013-11-08 15:22:13,886-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-grind.mcs.anl.gov:0 (quality: 143.0) submitting urn:job-0-1383759275080-1383759275081: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-08 15:22:13,886-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 143.0) submitting urn:job-0-1383759275080-1383759275081: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-08 15:22:13,887-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-2013-11-08 15:22:13,887-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-2013-11-08 15:22:13,893-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7618, JOBSTATUS) unregistering (send)
-2013-11-08 15:22:13,893-0600 INFO RequestHandler Handler(tag: 7618, JOBSTATUS) unregistering (send)
-Timeout check started for TCPChannel [type: server, contact: unknown]
-2013-11-08 15:22:13,896-0600 INFO AbstractCoasterChannel Timeout check started for TCPChannel [type: server, contact: unknown]
-(0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:38668] for addition
-2013-11-08 15:22:13,896-0600 INFO Multiplexer (0) Scheduling TCPChannel [type: server, contact: 127.0.0.1:38668] for addition
-Channel started: TCPChannel [type: server, contact: 127.0.0.1:38668]
-2013-11-08 15:22:13,896-0600 INFO AbstractTCPChannel Channel started: TCPChannel [type: server, contact: 127.0.0.1:38668]
-Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f27
-2013-11-08 15:22:13,899-0600 INFO ChannelConfigurationHandler Channel id: channel-3-u29cf5f3a-1422e7ae15d--7f27
-MetaChannel [context: cps-38668, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:38668]
-2013-11-08 15:22:13,899-0600 INFO MetaChannel MetaChannel [context: cps-38668, boundTo: null] binding to TCPChannel [type: server, contact: 127.0.0.1:38668]
-Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-2013-11-08 15:22:13,899-0600 INFO RequestHandler Handler(tag: 42564, CHANNELCONFIG) unregistering (send)
-Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-2013-11-08 15:22:13,939-0600 INFO RequestHandler Handler(tag: 42565, CONFIGSERVICE) unregistering (send)
-Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-08 15:22:13,939-0600 INFO RequestHandler Handler(tag: 31105, SUBMITJOB) unregistering (send)
-2013-11-08 15:22:13,987-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-crush.mcs.anl.gov:0 (quality: 56.0) submitting urn:job-1-1383759275083-1383759275084: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-08 15:22:13,987-0600 INFO Cpu crush.mcs.anl.gov:0 (quality: 56.0) submitting urn:job-1-1383759275083-1383759275084: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-08 15:22:13,992-0600 DEBUG Cpu Active workerid=crush.mcs.anl.gov:000000
-Handler(tag: 143, JOBSTATUS) unregistering (send)
-2013-11-08 15:22:13,993-0600 INFO RequestHandler Handler(tag: 143, JOBSTATUS) unregistering (send)
-Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-08 15:22:14,040-0600 INFO RequestHandler Handler(tag: 42566, SUBMITJOB) unregistering (send)
-2013-11-08 15:22:15,919-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-08 15:22:15,919-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 15:22:15,919-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 15:22:15,919-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-Channel config: null -> {keepalive=null}
-2013-11-08 15:22:15,919-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:38666] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 15:22:15,919-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:38666] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 15:22:15,919-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 1
-null: Channel shut down
-2013-11-08 15:22:15,919-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-38666, boundTo: TCPChannel [type: server, contact: 127.0.0.1:38666]] binding to NullChannel
-2013-11-08 15:22:15,920-0600 INFO MetaChannel MetaChannel [context: cps-38666, boundTo: TCPChannel [type: server, contact: 127.0.0.1:38666]] binding to NullChannel
-Handler(tag: 7619, JOBSTATUS) unregistering (send)
-2013-11-08 15:22:15,919-0600 INFO RequestHandler Handler(tag: 7619, JOBSTATUS) unregistering (send)
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:38666]
-2013-11-08 15:22:15,920-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:38666]
-grind.mcs.anl.gov:0 (quality: 144.0) submitting urn:job-2-1383759275086-1383759275087: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-2013-11-08 15:22:15,920-0600 INFO Cpu grind.mcs.anl.gov:0 (quality: 144.0) submitting urn:job-2-1383759275086-1383759275087: /bin/bash /homes/yadunand/bin/exm-trunk/sfw/turbine/branches/issue-503/code/tests/coasters/wrapper.sh
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 15:22:15,920-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-08 15:22:15,920-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 15:22:15,920-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-08 15:22:15,920-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 15:22:15,920-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-08 15:22:15,921-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 15:22:15,921-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-08 15:22:15,921-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 15:22:15,921-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-08 15:22:15,921-0600 INFO ChannelManager Channel exception handled
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 15:22:15,921-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel config: null -> {keepalive=null}
-2013-11-08 15:22:15,921-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:38668] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 15:22:15,921-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:38668] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-null: Channel shut down
-2013-11-08 15:22:15,921-0600 INFO AbstractTCPChannel null: Channel shut down
-MetaChannel [context: cps-38668, boundTo: TCPChannel [type: server, contact: 127.0.0.1:38668]] binding to NullChannel
-2013-11-08 15:22:15,921-0600 INFO MetaChannel MetaChannel [context: cps-38668, boundTo: TCPChannel [type: server, contact: 127.0.0.1:38668]] binding to NullChannel
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:38668]
-2013-11-08 15:22:15,921-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:38668]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 15:22:15,921-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-08 15:22:15,921-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 15:22:15,921-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-08 15:22:15,922-0600 INFO ChannelManager Channel exception handled
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-08 15:22:15,925-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:42)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.notifyListeners(TaskImpl.java:240)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:228)
- at org.globus.cog.abstraction.impl.common.task.TaskImpl.setStatus(TaskImpl.java:257)
- at org.globus.cog.abstraction.coaster.service.job.manager.Cpu.replyReceived(Cpu.java:397)
- at org.globus.cog.coaster.commands.Command.receiveCompleted(Command.java:181)
- at org.globus.cog.abstraction.impl.execution.coaster.SubmitJobCommand.receiveCompleted(SubmitJobCommand.java:227)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleReply(AbstractCoasterChannel.java:425)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:234)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-08 15:22:15,925-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-08 15:22:15,925-0600 DEBUG Cpu Active workerid=grind.mcs.anl.gov:000000
-Handler(tag: 7620, JOBSTATUS) unregistering (send)
-2013-11-08 15:22:15,925-0600 INFO RequestHandler Handler(tag: 7620, JOBSTATUS) unregistering (send)
-Channel is closed. Removing.
-2013-11-08 15:22:15,926-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-08 15:22:15,926-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-08 15:22:15,926-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:38668] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-08 15:22:15,926-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:38668] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-08 15:22:15,926-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:38668]
-2013-11-08 15:22:15,926-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:38668]
-Channel is closed. Removing.
-2013-11-08 15:22:15,926-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-08 15:22:15,926-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-08 15:22:15,926-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:38666] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-08 15:22:15,926-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:38666] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-08 15:22:15,926-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:38666]
-2013-11-08 15:22:15,926-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:38666]
-2013-11-08 15:22:16,178-0600 DEBUG Cpu Completed
-crush.mcs.anl.gov:0 jobTerminated
-2013-11-08 15:22:16,178-0600 INFO Cpu crush.mcs.anl.gov:0 jobTerminated
-Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 15:22:16,178-0600 INFO Sender Channel IOException
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 15:22:16,178-0600 DEBUG Cpu ready for work: block=crush.mcs.anl.gov id=0
-Channel config: null -> {keepalive=null}
-2013-11-08 15:22:16,178-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:38667] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 15:22:16,178-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:38667] handling channel exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Handler(tag: 144, JOBSTATUS) unregistering (send)
-2013-11-08 15:22:16,178-0600 INFO RequestHandler Handler(tag: 144, JOBSTATUS) unregistering (send)
-null: Channel shut down
-2013-11-08 15:22:16,178-0600 INFO AbstractTCPChannel null: Channel shut down
-2013-11-08 15:22:16,178-0600 DEBUG Cpu requesting work: block=crush.mcs.anl.gov id=0 Cpus sleeping: 1
-MetaChannel [context: cps-38667, boundTo: TCPChannel [type: server, contact: 127.0.0.1:38667]] binding to NullChannel
-2013-11-08 15:22:16,178-0600 INFO MetaChannel MetaChannel [context: cps-38667, boundTo: TCPChannel [type: server, contact: 127.0.0.1:38667]] binding to NullChannel
-2013-11-08 15:22:16,178-0600 DEBUG Cpu crush.mcs.anl.gov:0 sleeping
-Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:38667]
-2013-11-08 15:22:16,178-0600 INFO Multiplexer Unregistering channel TCPChannel [type: server, contact: 127.0.0.1:38667]
-Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 15:22:16,178-0600 INFO ChannelContext Notifying commands and handlers about exception
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-08 15:22:16,178-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 15:22:16,178-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-08 15:22:16,178-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 15:22:16,179-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-08 15:22:16,179-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 15:22:16,179-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-08 15:22:16,179-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-2013-11-08 15:22:16,179-0600 WARN TaskNotifier Client could not properly process notification: null
-java.net.SocketException: Broken pipe
- at java.net.SocketOutputStream.socketWrite0(Native Method)
- at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
- at java.net.SocketOutputStream.write(SocketOutputStream.java:141)
- at org.globus.cog.coaster.channels.Sender.send(Sender.java:130)
- at org.globus.cog.coaster.channels.Sender.run(Sender.java:66)
-Channel exception handled
-2013-11-08 15:22:16,179-0600 INFO ChannelManager Channel exception handled
-Channel is closed. Removing.
-2013-11-08 15:22:16,187-0600 INFO Multiplexer Channel is closed. Removing.
-Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-08 15:22:16,187-0600 INFO Multiplexer Exception in channel step
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel config: null -> {keepalive=null}
-2013-11-08 15:22:16,187-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:38667] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-2013-11-08 15:22:16,187-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:38667] handling channel exception
-java.io.IOException: Stream closed.
- at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:435)
- at java.net.SocketInputStream.available(SocketInputStream.java:234)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.step(AbstractStreamCoasterChannel.java:105)
- at org.globus.cog.coaster.channels.Multiplexer.run(Multiplexer.java:70)
-Channel already shut down
-2013-11-08 15:22:16,187-0600 INFO ChannelManager Channel already shut down
-Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:38667]
-2013-11-08 15:22:16,187-0600 INFO Multiplexer Removing faulty channel TCPChannel [type: server, contact: 127.0.0.1:38667]
-Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-08 15:22:17,936-0600 WARN TaskNotifier Failed to send task notification
-org.globus.cog.coaster.channels.ChannelException: Channel died and no contact available
- at org.globus.cog.coaster.channels.ChannelManager.connect(ChannelManager.java:234)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:255)
- at org.globus.cog.coaster.channels.ChannelManager.reserveChannel(ChannelManager.java:226)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.sendStatus(TaskNotifier.java:56)
- at org.globus.cog.abstraction.coaster.service.job.manager.TaskNotifier.statusChanged(TaskNotifier.java:48)
- at org.globus.cog.abstraction.impl.execution.coaster.NotificationManager.notificationReceived(NotificationManager.java:99)
- at org.globus.cog.abstraction.coaster.service.local.JobStatusHandler.requestComplete(JobStatusHandler.java:56)
- at org.globus.cog.coaster.handlers.RequestHandler.receiveCompleted(RequestHandler.java:88)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.handleRequest(AbstractCoasterChannel.java:527)
- at org.globus.cog.coaster.channels.AbstractStreamCoasterChannel.stepNIO(AbstractStreamCoasterChannel.java:238)
- at org.globus.cog.coaster.channels.NIOMultiplexer.loop(NIOMultiplexer.java:97)
- at org.globus.cog.coaster.channels.NIOMultiplexer.run(NIOMultiplexer.java:56)
-2013-11-08 15:22:17,936-0600 DEBUG Cpu Completed
-grind.mcs.anl.gov:0 jobTerminated
-2013-11-08 15:22:17,936-0600 INFO Cpu grind.mcs.anl.gov:0 jobTerminated
-2013-11-08 15:22:17,937-0600 DEBUG Cpu ready for work: block=grind.mcs.anl.gov id=0
-2013-11-08 15:22:17,937-0600 DEBUG Cpu requesting work: block=grind.mcs.anl.gov id=0 Cpus sleeping: 2
-2013-11-08 15:22:17,937-0600 DEBUG Cpu grind.mcs.anl.gov:0 sleeping
-Handler(tag: 7621, JOBSTATUS) unregistering (send)
-2013-11-08 15:22:17,937-0600 INFO RequestHandler Handler(tag: 7621, JOBSTATUS) unregistering (send)
-No streams
-2013-11-08 15:22:19,155-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:22:29,163-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:22:39,171-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-2013-11-08 15:22:43,028-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:22:48,403-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:22:49,179-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:22:59,187-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:23:09,195-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:23:19,203-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 15:23:19,501-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:23:25,721-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:23:29,211-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:23:39,219-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:23:49,227-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 15:23:55,975-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 15:23:59,235-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:24:03,039-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:24:09,244-0600 INFO Multiplexer No streams
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-152213.858, now=131108-152414.707, channel=TCPChannel [type: server, contact: 127.0.0.1:38666]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 15:24:14,707-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-152213.858, now=131108-152414.707, channel=TCPChannel [type: server, contact: 127.0.0.1:38666]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-08 15:24:14,708-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-152213.858, now=131108-152414.707, channel=TCPChannel [type: server, contact: 127.0.0.1:38666]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 15:24:14,708-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-152213.858, now=131108-152414.707, channel=TCPChannel [type: server, contact: 127.0.0.1:38666]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-08 15:24:14,708-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-152213.858, now=131108-152414.707, channel=TCPChannel [type: server, contact: 127.0.0.1:38666]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 15:24:14,708-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-152213.858, now=131108-152414.707, channel=TCPChannel [type: server, contact: 127.0.0.1:38666]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-08 15:24:14,708-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-152213.858, now=131108-152414.707, channel=TCPChannel [type: server, contact: 127.0.0.1:38666]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 15:24:14,708-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-152213.858, now=131108-152414.707, channel=TCPChannel [type: server, contact: 127.0.0.1:38666]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-08 15:24:14,708-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-152213.858, now=131108-152414.707, channel=TCPChannel [type: server, contact: 127.0.0.1:38666]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 15:24:14,708-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-152213.858, now=131108-152414.707, channel=TCPChannel [type: server, contact: 127.0.0.1:38666]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-08 15:24:14,708-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:38666] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-152213.858, now=131108-152414.707, channel=TCPChannel [type: server, contact: 127.0.0.1:38666]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 15:24:14,708-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:38666] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-152213.858, now=131108-152414.707, channel=TCPChannel [type: server, contact: 127.0.0.1:38666]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-08 15:24:14,708-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-152213.939, now=131108-152414.816, channel=TCPChannel [type: server, contact: 127.0.0.1:38667]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 15:24:14,816-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-152213.939, now=131108-152414.816, channel=TCPChannel [type: server, contact: 127.0.0.1:38667]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-08 15:24:14,816-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-152213.939, now=131108-152414.816, channel=TCPChannel [type: server, contact: 127.0.0.1:38667]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 15:24:14,816-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-152213.939, now=131108-152414.816, channel=TCPChannel [type: server, contact: 127.0.0.1:38667]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 2, JOBSTATUS)
-2013-11-08 15:24:14,816-0600 INFO ChannelContext => Command(tag: 2, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-152213.939, now=131108-152414.816, channel=TCPChannel [type: server, contact: 127.0.0.1:38667]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 15:24:14,816-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-152213.939, now=131108-152414.816, channel=TCPChannel [type: server, contact: 127.0.0.1:38667]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 3, JOBSTATUS)
-2013-11-08 15:24:14,816-0600 INFO ChannelContext => Command(tag: 3, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-152213.939, now=131108-152414.816, channel=TCPChannel [type: server, contact: 127.0.0.1:38667]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 15:24:14,816-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-152213.939, now=131108-152414.816, channel=TCPChannel [type: server, contact: 127.0.0.1:38667]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 4, JOBSTATUS)
-2013-11-08 15:24:14,817-0600 INFO ChannelContext => Command(tag: 4, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-152213.939, now=131108-152414.816, channel=TCPChannel [type: server, contact: 127.0.0.1:38667]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 15:24:14,817-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-152213.939, now=131108-152414.816, channel=TCPChannel [type: server, contact: 127.0.0.1:38667]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-08 15:24:14,817-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:38667] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-152213.939, now=131108-152414.816, channel=TCPChannel [type: server, contact: 127.0.0.1:38667]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 15:24:14,817-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:38667] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-152213.939, now=131108-152414.816, channel=TCPChannel [type: server, contact: 127.0.0.1:38667]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-08 15:24:14,817-0600 INFO ChannelManager Channel already shut down
-Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-152214.040, now=131108-152414.896, channel=TCPChannel [type: server, contact: 127.0.0.1:38668]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 15:24:14,896-0600 INFO ChannelContext Notifying commands and handlers about exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-152214.040, now=131108-152414.896, channel=TCPChannel [type: server, contact: 127.0.0.1:38668]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-=> Command(tag: 1, JOBSTATUS)
-2013-11-08 15:24:14,896-0600 INFO ChannelContext => Command(tag: 1, JOBSTATUS)
-Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-152214.040, now=131108-152414.896, channel=TCPChannel [type: server, contact: 127.0.0.1:38668]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 15:24:14,896-0600 WARN TaskNotifier Client could not properly process notification: null
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-152214.040, now=131108-152414.896, channel=TCPChannel [type: server, contact: 127.0.0.1:38668]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel config: null -> {keepalive=null}
-2013-11-08 15:24:14,896-0600 INFO AbstractCoasterChannel Channel config: null -> {keepalive=null}
-TCPChannel [type: server, contact: 127.0.0.1:38668] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-152214.040, now=131108-152414.896, channel=TCPChannel [type: server, contact: 127.0.0.1:38668]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-2013-11-08 15:24:14,896-0600 INFO ChannelManager TCPChannel [type: server, contact: 127.0.0.1:38668] handling channel exception
-org.globus.cog.coaster.TimeoutException: Channel timed out. lastTime=131108-152214.040, now=131108-152414.896, channel=TCPChannel [type: server, contact: 127.0.0.1:38668]
- at org.globus.cog.coaster.channels.AbstractCoasterChannel.checkTimeouts(AbstractCoasterChannel.java:133)
- at org.globus.cog.coaster.channels.AbstractCoasterChannel$1.run(AbstractCoasterChannel.java:124)
- at java.util.TimerThread.mainLoop(Timer.java:534)
- at java.util.TimerThread.run(Timer.java:484)
-Channel already shut down
-2013-11-08 15:24:14,896-0600 INFO ChannelManager Channel already shut down
-No streams
-2013-11-08 15:24:19,252-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:24:29,260-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:24:32,449-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:24:39,268-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-2013-11-08 15:24:40,358-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-No streams
-2013-11-08 15:24:49,277-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:24:59,285-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 15:25:08,923-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 15:25:09,293-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-2013-11-08 15:25:17,676-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-No streams
-2013-11-08 15:25:19,301-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:25:29,309-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:25:39,318-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 15:25:45,397-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 15:25:49,326-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:25:54,993-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:25:59,334-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:26:09,342-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:26:19,351-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 15:26:21,871-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 15:26:29,359-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:26:32,312-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:26:39,367-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:26:49,375-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:26:58,345-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:26:59,384-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:27:09,392-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:27:09,631-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:27:19,401-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:27:29,409-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 15:27:34,820-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 15:27:39,418-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 15:27:46,949-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 15:27:49,426-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:27:59,434-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:28:09,443-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 15:28:11,294-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 15:28:19,451-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-2013-11-08 15:28:24,267-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-No streams
-2013-11-08 15:28:29,460-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:28:39,468-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 15:28:47,768-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 15:28:49,479-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:28:59,487-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-2013-11-08 15:29:01,585-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-No streams
-2013-11-08 15:29:09,499-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:29:19,509-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:29:24,241-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:29:29,517-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:29:38,902-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:29:39,528-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:29:49,537-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:29:59,555-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 15:30:00,715-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 15:30:09,559-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-2013-11-08 15:30:16,221-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-No streams
-2013-11-08 15:30:19,570-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:30:29,579-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 15:30:37,190-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 15:30:39,599-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:30:49,609-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-2013-11-08 15:30:53,539-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-No streams
-2013-11-08 15:30:59,620-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:31:09,631-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:31:13,664-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:31:19,641-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:31:29,651-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-2013-11-08 15:31:30,857-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-No streams
-2013-11-08 15:31:39,661-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:31:49,671-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 15:31:50,139-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 15:31:59,683-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:32:08,174-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:32:09,695-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:32:19,704-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 15:32:26,613-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 15:32:29,714-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:32:39,723-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-2013-11-08 15:32:45,493-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-No streams
-2013-11-08 15:32:49,732-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:32:59,740-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 15:33:03,087-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 15:33:09,748-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:33:19,756-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 503ms
-2013-11-08 15:33:23,312-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 503ms
-No streams
-2013-11-08 15:33:29,767-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 15:33:39,561-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 15:33:39,780-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:33:49,797-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:33:59,807-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:34:00,129-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:34:09,818-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 15:34:16,035-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 15:34:19,828-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:34:29,836-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:34:37,447-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:34:39,844-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:34:49,853-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 15:34:52,509-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 15:34:59,862-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:35:09,871-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:35:14,765-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:35:19,882-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:35:28,982-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:35:29,891-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:35:39,899-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:35:49,907-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:35:52,083-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:35:59,915-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 15:36:05,457-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 15:36:09,923-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:36:19,932-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-2013-11-08 15:36:29,402-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-No streams
-2013-11-08 15:36:29,940-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:36:39,949-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 15:36:41,931-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 15:36:49,959-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:36:59,969-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:37:06,720-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:37:09,978-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 15:37:18,405-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 15:37:19,988-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:37:29,998-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:37:40,007-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:37:44,038-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:37:50,015-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 15:37:54,880-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 15:38:00,025-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:38:10,035-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:38:20,045-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-2013-11-08 15:38:21,357-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-No streams
-2013-11-08 15:38:30,054-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 15:38:31,353-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 15:38:40,064-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:38:50,074-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-2013-11-08 15:38:58,675-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-No streams
-2013-11-08 15:39:00,082-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:39:07,827-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:39:10,091-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:39:20,100-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:39:30,109-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:39:35,992-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:39:40,118-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-2013-11-08 15:39:44,302-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 2ms
-No streams
-2013-11-08 15:39:50,128-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:40:00,138-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:40:10,146-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:40:13,310-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:40:20,155-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:40:20,775-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:40:30,167-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:40:40,177-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:40:50,187-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:40:50,629-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:40:57,249-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:41:00,197-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:41:10,207-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:41:20,216-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-2013-11-08 15:41:27,948-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-No streams
-2013-11-08 15:41:30,224-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:41:33,723-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:41:40,232-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:41:50,240-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:42:00,248-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-2013-11-08 15:42:05,266-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:42:10,197-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:42:10,255-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:42:20,263-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:42:30,271-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:42:40,279-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-2013-11-08 15:42:42,584-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 15:42:46,672-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 15:42:50,287-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:43:00,294-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:43:10,302-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:43:19,901-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:43:20,310-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:43:23,146-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:43:30,318-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:43:40,326-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:43:50,334-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-2013-11-08 15:43:57,220-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 15:43:59,620-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 15:44:00,342-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:44:10,349-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:44:20,358-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:44:30,365-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:44:34,537-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:44:36,094-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:44:40,373-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:44:50,381-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:45:00,389-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:45:10,397-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:45:11,855-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 15:45:12,568-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-No streams
-2013-11-08 15:45:20,404-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:45:30,412-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:45:40,420-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:45:49,173-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1029ms
-2013-11-08 15:45:50,070-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1029ms
-No streams
-2013-11-08 15:45:50,428-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:46:00,436-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:46:10,450-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:46:20,460-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 421ms
-2013-11-08 15:46:25,936-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 421ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-2013-11-08 15:46:26,492-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 2ms
-No streams
-2013-11-08 15:46:30,468-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:46:40,476-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:46:50,485-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:47:00,493-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-2013-11-08 15:47:01,989-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 0ms
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:47:03,809-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:47:10,501-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:47:20,509-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:47:30,517-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-2013-11-08 15:47:38,464-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: -1ms, rtt: 1ms
-No streams
-2013-11-08 15:47:40,524-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 2869ms
-2013-11-08 15:47:43,995-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: grind.mcs.anl.gov-000000] up latency: -1ms, rtt: 2869ms
-No streams
-2013-11-08 15:47:50,532-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:48:00,540-0600 INFO Multiplexer No streams
-No streams
-2013-11-08 15:48:10,548-0600 INFO Multiplexer No streams
-TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
-2013-11-08 15:48:14,938-0600 INFO HeartBeatCommand TCPChannel [type: server, contact: crush.mcs.anl.gov-000000] up latency: 0ms, rtt: 1ms
Index: modules/provider-coaster-c-client/tests/coasters/tc.data
===================================================================
--- modules/provider-coaster-c-client/tests/coasters/tc.data (revision 3830)
+++ modules/provider-coaster-c-client/tests/coasters/tc.data (working copy)
@@ -5,3 +5,4 @@
persistent-coasters cat /bin/cat null null null
persistent-coasters cat /bin/cat null null null
persistent-coasters cat /bin/cat null null null
+persistent-coasters cat /bin/cat null null null
More information about the Swift-commit
mailing list